Retrieval-augmented generation (RAG) is a technique in which an AI system first searches a collection of documents for passages relevant to a question, then gives those passages to a language model and asks it to answer from them, ideally citing where each point came from. The model supplies the language; your content supplies the facts.
That makes RAG one of the most direct ways an archive becomes useful to AI. A newsroom can answer questions from decades of reporting, and an educational publisher can build a study assistant that stays within its own textbooks. In every case, answer quality depends heavily on the quality, structure and metadata of the content underneath.
This guide covers where RAG came from, each stage of the pipeline, why retrieval reduces hallucination, what content owners need to get right, how to evaluate a RAG system, and the most common failure modes.
Key takeaways
- RAG combines a search step over your own documents with a language model that writes answers grounded in the passages it retrieved.
- Retrieval reduces hallucination and lets you change what the system knows by updating the index, but it does not guarantee correct answers.
- Most RAG quality problems start upstream, in messy text, poor chunking, missing metadata and stale or duplicated content.
- Evaluate retrieval and generation separately: first check whether the right passages were found, then check whether the answer is faithful to them.
- Permissions, freshness and source identifiers need to be designed into the index from the start rather than added after launch.
Where RAG comes from
The term comes from a 2020 paper by Patrick Lewis and colleagues, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, presented at NeurIPS 2020. The authors started from a known weakness: large language models store a great deal of factual knowledge in their parameters, but they cannot access it precisely, show where an answer came from, or easily update what they know.
Their solution paired two kinds of memory. The parametric memory was a pre-trained sequence-to-sequence model (BART-large, around 400 million parameters). The non-parametric memory was a dense vector index of Wikipedia: the December 2018 dump, split into disjoint 100-word chunks, about 21 million in total, searched with a neural retriever called Dense Passage Retrieval (DPR). The approach set the state of the art on three open-domain question-answering tasks and produced language the authors described as more specific, diverse and factual than a comparable model without retrieval.
One experiment in the paper matters especially for content owners. The researchers built a second index from a December 2016 Wikipedia dump and asked the model 'Who is {position}?' for 82 world leaders who had changed between 2016 and 2018. When the index matched the year being asked about, the model answered about 70% (2016) and 68% (2018) correctly. When they were mismatched, accuracy fell to 12% and 4%. The conclusion: you can update a RAG system's knowledge by replacing its index, without retraining the model.
Many of today's systems use general-purpose chat models rather than BART, and retrieval often mixes keyword and vector search, but the architecture is the same.
The RAG pipeline, step by step
A production RAG system has two halves: an offline pipeline that prepares and indexes content, and an online pipeline that runs every time someone asks a question.
1. Ingest
Content is collected from wherever it lives (CMS exports, PDFs, XML, scanned pages, transcripts) and converted to clean text with its structure preserved: headings, paragraphs, tables, captions, bylines and dates. A scanned page with poor OCR, or a two-column PDF read straight across both columns, produces text that no retriever can rescue.
2. Chunk
Documents are split into passages small enough to retrieve precisely and to fit, several at a time, into the model's context. Lewis et al. used fixed 100-word chunks; many current systems split along headings and paragraphs instead. The trade-offs are covered in our guide to chunking long documents.
3. Embed
Each chunk is converted by an embedding model into a vector, a list of numbers that places passages with similar meaning close together. This is what lets a question about 'council housing shortages' find an article about 'a lack of social homes'. See embeddings and vector search, explained without the maths.
4. Index
Vectors are stored in an index built for fast approximate nearest-neighbour search, alongside the chunk text, a stable document ID and metadata such as date, section, language and access level. Many systems also build a keyword index, typically using BM25, over the same chunks.
5. Retrieve
At question time the query is embedded and the index returns the closest chunks, narrowed by metadata filters (only articles after 2020, only collections this user may see). Hybrid systems run keyword and vector search together and merge the results.
6. Re-rank
A first-pass retriever is tuned for speed, so a second model often re-scores the top candidates more carefully by reading the query and each passage together. Nogueira and Cho showed the value of this in 2019: using BERT as a passage re-ranker, they beat the previous best MS MARCO passage result by 27% (relative) on MRR@10. In an evaluation Anthropic published in 2024, combining hybrid search, a short generated context for each chunk and re-ranking 150 candidates down to 20 reduced the share of relevant chunks missing from the top 20 from 5.7% to 1.9%.
7. Generate with citations
The top passages are placed in a prompt with the question and instructions: answer only from the sources provided, cite them, and say so when they do not contain the answer. Because each chunk carries its document ID, title, date and URL, readers can see where each claim came from. Lewis et al. noted this benefit: a retrieval index is human-readable and can be inspected.
| Stage | What happens | Typical problem |
|---|---|---|
| Ingest | Files become clean, structured text | OCR errors, broken reading order, lost tables |
| Chunk | Text is split into retrievable passages | Chunks that cut ideas in half or lose their heading |
| Embed | Chunks become vectors | A model poorly suited to the language or subject |
| Index | Vectors, text and metadata are stored | Missing dates, permissions or IDs |
| Retrieve | Candidate chunks are found | Exact names and codes missed by vector-only search |
| Re-rank | Candidates are re-ordered by relevance | Skipped to save time, so weak matches reach the model |
| Generate | A cited answer is written | Claims that go beyond the sources |
Why retrieval reduces hallucination and keeps answers current
A language model answering from memory produces the most plausible continuation of what it has seen. When it lacks a fact, it can still produce a fluent, confident and wrong answer. Supplying the relevant passage turns the task from recall into reading comprehension. Shuster and colleagues applied retrieval to knowledge-grounded conversation in 2021 and found, in human evaluations, that it substantially reduced knowledge hallucination in state-of-the-art chatbots.
Retrieval also separates knowledge from the model, so updating the index updates the answers. A story published this morning can be retrieved as soon as it is indexed, while retraining a model is slow and expensive.
Three caveats keep expectations realistic:
- Retrieval can fail. If the right passage is not retrieved, the model either says it does not know or, worse, answers from memory anyway.
- Position in the prompt matters. Liu et al. ('Lost in the Middle', TACL 2023) found that models use information best at the beginning or end of the context and noticeably worse in the middle, even models designed for long contexts. Extra marginal passages are not a free safety margin.
- Sources can be wrong or out of date. RAG repeats what your archive says, including errors whose corrections were never linked to the original.
What content owners must get right
Most of the levers that decide RAG quality sit with whoever owns the content, not with whoever picks the model.
Clean, structured text
Extract text in the right reading order, keep headings and table structure, remove navigation menus and repeated footers, and fix OCR errors where they matter. Boilerplate that appears on every page matches many queries weakly and crowds out useful passages.
Metadata that supports filtering
Filters only work on fields that exist and are consistent: publication date, content type, subject, language, author, region, curriculum level. A question such as 'what did we report about the bypass before it opened?' needs reliable dates. The guide to metadata for publishers covers the main types.
Permissions-aware retrieval
If some content is restricted (subscriber-only articles, embargoed reports, internal documents), apply access control in the retrieval step, before any passage reaches the model. Store access attributes on every chunk, filter each search by the user's entitlements, and re-index when permissions change. A prompt telling the model not to reveal restricted text is not access control.
Freshness, corrections and versions
Index new content promptly, remove withdrawn content, link corrections and new editions to what they replace, and record which version is current so the retriever can prefer it.
Duplicates and stable identifiers
Syndicated copies and near-identical versions can fill the top results with the same passage; see finding duplicates and near-duplicates. Give every document a stable ID and every chunk a reference back to it, so citations survive re-indexing.
Evaluating a RAG system
Evaluate retrieval and generation separately, because they fail for different reasons and are fixed in different places.
Start with a test set: a few hundred realistic questions from search logs or from people who know the archive, each paired with the passage that answers it. Include questions the archive cannot answer.
| Metric | Question it answers | How it is usually measured |
|---|---|---|
| Hit rate at k | Did at least one relevant chunk appear in the top k results? | Labelled question and passage pairs |
| Mean reciprocal rank (MRR) | How high did the first relevant chunk rank? | The same labelled set |
| Context relevance | How much of the retrieved text was actually useful? | Reviewer or model judgement |
| Faithfulness (groundedness) | Is every claim supported by the retrieved passages? | Claim-by-claim check by reviewers or a model |
| Answer relevance | Does the answer address the question asked? | Reviewer rating or automated scoring |
| Abstention | Does the system say so when the archive lacks the answer? | Questions with no answer in the collection |
Frameworks can automate parts of this. Ragas (Es et al., 2023), for example, proposed metrics for context relevance, faithfulness and answer relevance that work without human-written reference answers. Automated judges are useful for tracking changes between versions, but calibrate them against human judgements first.
Evaluation does not end at launch. Barnett et al., drawing on RAG case studies in research, education and biomedicine, concluded that a RAG system can only realistically be validated in operation and that its robustness develops over time. Log questions, review a sample of answers regularly, and add every failure you find to the test set.
Common failure modes
Barnett et al. identified seven failure points when engineering RAG systems. They make a useful diagnostic checklist.
| Failure point | What happens | Where to look first |
|---|---|---|
| Missing content | The answer is not in the archive, and the system may answer anyway instead of saying it does not know | Coverage gaps; instructions to abstain |
| Missed the top-ranked documents | The answer exists but does not rank in the top k | Hybrid search, re-ranking, chunking |
| Not in context | A relevant document is retrieved but dropped when the prompt is assembled | Context budget, ordering, duplicate removal |
| Not extracted | The answer is in the prompt, but the model fails to use it | Noisy or conflicting passages |
| Wrong format | The model ignores a format instruction, such as a table | Prompt design and output validation |
| Incorrect specificity | The answer is too general or too narrow for the user | Question clarification, chunk granularity |
| Incomplete | The answer is correct but leaves out available information | Multi-part questions, retrieval breadth |
Content teams will recognise a few more: superseded material outranking the current version because nothing marks it as superseded, exact identifiers such as ISBNs or case numbers missed by vector-only search, and restricted text leaking because access was checked after retrieval rather than during it. Each is a data problem before it is a model problem.
How Druid Learning helps
Druid Learning is an AI archival processing platform, based in Dublin and New York, that turns content archives into AI-ready datasets. It handles content in 140+ languages and formats including OCR scans, PDF, XML, IDML, INDD, EPUB, WAV, MP3 and MP4, and Archive Optimization turns an archive in any format into an indexed, labelled, AI-ready dataset. For NPR, 2M+ assets were processed with 20+ labels across 8 dimensions, such as topic, location and article type, supporting archive search by label and suggested related stories for journalists. Content owners preparing an archive for AI companies can also work through Data Partnerships.
Frequently asked questions
Is RAG the same as fine-tuning?
No. Fine-tuning changes a model's weights by training it further on examples, which suits teaching a style, a format or a specialised task. RAG leaves the model unchanged and supplies relevant documents at the moment a question is asked. RAG is usually the better fit when facts change often, when answers must cite sources, or when access to content differs between users. The two approaches can also be combined.
Does RAG stop AI models from hallucinating?
RAG reduces hallucination but does not eliminate it. Research has found that retrieval substantially reduces invented facts compared with answering from memory, but a RAG system can still retrieve the wrong passage, miss the right one, or add claims its sources do not support. Requiring citations, measuring faithfulness and instructing the model to say when its sources lack the answer all reduce the remaining risk.
How much content does a RAG system need?
There is no minimum. RAG can work over a few hundred documents or many millions; the original 2020 system searched about 21 million Wikipedia passages. Volume matters less than coverage and condition: the content should answer the questions users actually ask, be cleanly extracted, carry accurate metadata such as dates and access levels, and not be dominated by duplicates.
What is hybrid search in a RAG system?
Hybrid search runs keyword search, usually based on BM25, alongside vector search and merges the two result lists. Vector search finds passages with similar meaning even when the wording differs, while keyword search reliably finds exact names, codes and rare terms that embeddings can miss. Combining the two, often followed by a re-ranking step, generally finds relevant passages more reliably than vector search alone.
How often should a RAG index be updated?
As often as the underlying content changes in ways users care about. A newsroom may index new stories continuously, while a textbook publisher may re-index when a new edition is released. Updates should cover deletions, corrections and permission changes as well as new material, because a stale index will confidently cite content that has been withdrawn or superseded.
Sources
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al., NeurIPS 2020 (arXiv)
- Retrieval Augmentation Reduces Hallucination in Conversation — Shuster et al. (arXiv)
- Passage Re-ranking with BERT — Nogueira and Cho (arXiv)
- Introducing Contextual Retrieval — Anthropic
- Lost in the Middle: How Language Models Use Long Contexts — Liu et al., TACL (arXiv)
- Ragas: Automated Evaluation of Retrieval Augmented Generation — Es et al. (arXiv)
- Seven Failure Points When Engineering a Retrieval Augmented Generation System — Barnett et al. (arXiv)