How AI Models Use Data: Pre-training, Fine-tuning and Evaluation

Large language models use data in several separate stages, and each stage needs different content. Most of a model's knowledge comes from pre-training on very large collections of general text. Much smaller, carefully made datasets then teach it to follow instructions (fine-tuning) and to give the answers people prefer (preference training). Held-out evaluation data measures how well it works. Retrieval hands the model documents at the moment it answers, without changing the model at all.

For anyone who holds an archive, this distinction matters. A newspaper back catalogue, a set of textbooks and a database of expert-reviewed questions are useful in very different ways, depending on which stage they would feed.

This guide walks through each stage using figures from published model papers (GPT-3, LLaMA, Chinchilla, InstructGPT, LIMA and Llama 3). It explains what kind of content each stage uses and why quality and diversity keep coming up, then turns that into a practical checklist for content owners.

Key takeaways

• Pre-training uses hundreds of billions to trillions of tokens of general text, and it is where a model learns most of what it knows.
• In the InstructGPT and LIMA papers, fine-tuning and preference datasets range from 1,000 to about 33,000 prompts, and at that scale careful curation matters more than volume.
• Evaluation data has to stay out of training data, or benchmark scores stop meaning anything.
• Retrieval-augmented generation (RAG) puts content in front of a model when it answers, without training on it, which is why current, attributable content matters for AI search products.
• In both pre-training and fine-tuning, the papers covered here find that filtered, deduplicated and varied data works better than raw volume.

The stages at a glance

A token is the unit of text a model processes, usually a word or part of a word.

Pre-training: learning from very large general corpora

In pre-training, the model reads a huge body of text and learns to predict the next token. Nobody labels this data. What the model learns depends on what the corpus contains, how clean it is, and how often each source gets sampled.

The published data mixtures show how developers weigh their sources:

• GPT-3 (2020) had 175 billion parameters and was trained on 300 billion tokens. Filtered Common Crawl web data made up 60% of the training mix, WebText2 22%, two book corpora 8% each, and Wikipedia 3%. The authors said that "datasets we view as higher-quality are sampled more frequently". Wikipedia was seen 3.4 times during training, while the filtered Common Crawl data was seen less than half a time. Filtering cut Common Crawl from 45TB of compressed plain text to 570GB.
• LLaMA (2023) had a training set of about 1.4 trillion tokens of publicly available data: 67% CommonCrawl, 15% C4, 4.5% each of GitHub, Wikipedia and books, 2.5% arXiv and 2% Stack Exchange. LLaMA-13B outperformed GPT-3 on most benchmarks despite being ten times smaller.
• Llama 3 (2024) was trained on about 15 trillion multilingual tokens, against 1.8 trillion for Llama 2. The final mix was roughly 50% general knowledge, 25% maths and reasoning, 17% code and 8% multilingual text. Meta deduplicated the data at URL, document and line level, applied heuristic filters, and used classifiers to score document quality. Pre-training ended with a short phase that upsampled very high-quality data; the team found that small amounts of high-quality code and maths data at this stage can boost benchmark scores.

Why both scale and quality matter

The Chinchilla paper (2022) asked how much data a model of a given size needs. It found that, for compute-optimal training, model size and training tokens should be scaled equally: every doubling of model size should come with a doubling of training tokens. Chinchilla had 70 billion parameters and was trained on 1.4 trillion tokens, or 20 tokens per parameter. It used the same compute budget as the much larger Gopher (280B) and outperformed it, as well as GPT-3 (175B). The authors concluded that many large models of the time were "significantly undertrained".

The same paper also stressed quality. The authors speculated that scaling to larger datasets is "only beneficial when the data is high-quality", and called for collecting larger datasets "with a high focus on dataset quality". Developers want more tokens, but only good ones.

What this means for pre-training content

Three patterns repeat across these papers:

1. Edited, reference-grade text gets upweighted. LLaMA saw Wikipedia and books about twice each, and most other data about once.
2. Breadth of domain is deliberate. Code, maths, science papers and multilingual text are each given a share of the mix on purpose.
3. Cleaning is a large part of the work. Deduplication, filtering and quality scoring decide what survives. Our guide to deduplication in content archives covers the duplicate problem in detail.

Fine-tuning and instruction tuning: teaching a model how to respond

A pre-trained model continues text. It does not naturally answer questions or follow instructions. Supervised fine-tuning teaches it to do so by training on examples of prompts and good responses.

In the InstructGPT work, a team of about 40 contractors wrote demonstrations of the desired behaviour. The supervised fine-tuning set contained about 13,000 training prompts, some from real API usage and some written by the labellers.

LIMA (2023) showed how far curation can go. The researchers fine-tuned a 65-billion-parameter LLaMA model on only 1,000 carefully chosen prompts and responses. 750 came from community sites (Stack Exchange, wikiHow and Reddit's r/WritingPrompts); the authors wrote 200 themselves and adapted 50 from an existing instruction dataset. In human evaluations, LIMA's answers were equivalent to or preferred over GPT-4's in 43% of cases. The authors proposed that "a model's knowledge and capabilities are learnt almost entirely during pretraining". On this view, fine-tuning mostly teaches the model which style and format to use.

Content that suits fine-tuning:

• Clear question-and-answer pairs, worked examples and step-by-step explanations
• Material written to a consistent editorial standard, such as house style or a curriculum's way of explaining a concept
• Expert answers where correctness has been checked

Preference data and RLHF: learning what people prefer

Fine-tuning shows the model one good answer. Preference training shows it which of several answers people prefer. This stage usually uses reinforcement learning from human feedback (RLHF) or related methods such as direct preference optimisation (DPO).

InstructGPT described the classic three-step process:

1. Fine-tune the model on human-written demonstrations.
2. Have labellers rank between four and nine model outputs for each prompt, then train a reward model to predict which output people would prefer. The reward model dataset had about 33,000 training prompts.
3. Use reinforcement learning to optimise the model against that reward model, using about 31,000 prompts.

In the headline result, labellers preferred outputs from the 1.3-billion-parameter InstructGPT model to those from the 175-billion-parameter GPT-3, a model more than 100 times larger.

Llama 3's post-training ran six rounds, each of supervised fine-tuning followed by DPO. Annotators rated how strongly they preferred one response over another ("significantly better", "better", "slightly better" or "marginally better"), and could then edit the preferred response to improve it. For reward modelling and DPO, only pairs marked "significantly better" or "better" were used; pairs with similar responses were discarded. The developers trained only on the clearest judgements.

Most preference data is produced by annotators rather than taken from archives. Archives still contribute, though, by supplying realistic prompts, reference answers for annotators to check against, and domain material that experts use to judge correctness.

Evaluation and benchmark data: measuring what a model can do

Evaluation data measures a model's performance on tasks it was not trained on. Benchmarks are fixed test sets, such as GSM8K and MATH for mathematical reasoning, both used in Llama 3's evaluations.

The most important rule is separation. If test items leak into the training data, the model may have memorised the answers, and its score says little about real ability. This is called contamination. The GPT-3 authors acknowledged that because their data came from the web, "it is possible that our model was trained on some of our benchmark test sets". They had tried to remove overlaps in advance, but a bug meant the removal was only partial. To measure the damage, they built "clean" versions of each benchmark, excluding any example that shared a 13-gram with the pre-training data, and compared scores. In most cases performance changed only negligibly. Some benchmarks, though, were almost entirely present in the training data, and the authors left those results out.

Llama 3's developers took a preventive step: they included no training sets from commonly used benchmarks in their final high-quality training phase, so they could assess genuine generalisation.

For content owners, this makes certain material distinctive: carefully labelled, expert-verified items that have never been published online. Examples include exam-style questions with marked answers, fact-checked question sets, or specialist case material. Its value depends on staying out of training data.

Retrieval (RAG): using content without training on it

Retrieval-augmented generation keeps the model's weights fixed and gives it relevant documents at answer time. The 2020 RAG paper by Lewis and colleagues combined a model's parametric memory (what it learned in training) with a non-parametric memory: a dense vector index of Wikipedia, searched by a neural retriever. On language generation tasks, it produced "more specific, diverse and factual language" than a model working from its weights alone. The authors also noted that providing provenance for outputs and updating knowledge remained open problems for models that rely on weights alone.

That is why retrieval matters to publishers. A trained model's knowledge stops at its cut-off date: Llama 3's pre-training data had knowledge until the end of 2023. A retrieval index can be updated daily, and each answer can point to the document it drew on. In the RAG authors' words, models that hold knowledge only in their weights "cannot easily expand or revise their memory", whereas a document can simply be removed from an index.

Our explainer on retrieval-augmented generation covers how retrieval systems are built.

What this means for content owners

Before any conversation with an AI developer, it helps to work out which stage your content would serve best. A simple mapping:

A short preparation checklist:

1. Inventory what you hold: formats, date ranges, languages, rough volume.
2. Clean the text: fix OCR errors, strip boilerplate, keep headings and captions.
3. Deduplicate: syndicated copies and repeated templates reduce value at every stage.
4. Add metadata: dates, authors, subjects and content type let developers filter and mix your data.
5. Document the dataset: a datasheet describing sources, coverage and known gaps. See documenting a dataset.
6. Keep test material separate from anything delivered for training.

Common mistakes include assuming volume is all that matters, losing document structure during conversion, and supplying content without dates. Without dates, nobody can judge how fresh it is or how much of a time period it covers. For a fuller treatment of which qualities raise value, see what makes content valuable to AI developers.

How Druid Learning helps

Druid Learning is an AI archival processing platform, based in Dublin and New York, that turns content archives into AI-ready datasets. Whichever stage an AI developer has in mind, the groundwork is the same: clean, indexed, labelled content. Archive Optimization turns archives in any format into an indexed, labelled, AI-ready dataset; the platform processes formats including OCR scans, PDF, XML, IDML, INDD, EPUB, WAV, MP3 and MP4. It has processed 10M+ assets and handles content in 140+ languages. Through Data Partnerships, Druid prepares an archive, raw or structured, and connects content owners with AI companies.

Frequently asked questions

What is the difference between training data and retrieval data?

Training data is used before a model is released, and it changes the model's internal weights: the model learns from it. Retrieval data is kept in a searchable index and handed to the model at the moment it answers a question, so the model itself does not change. Retrieval content can be updated continuously and cited in answers. What a model learned in training stays fixed at its training cut-off date.

How much data does it take to fine-tune a large language model?

Far less than pre-training. Two well-known published examples are about 13,000 prompts for InstructGPT's supervised fine-tuning and only 1,000 curated examples in the LIMA study, which fine-tuned a 65-billion-parameter model. Pre-training, by contrast, uses hundreds of billions to trillions of tokens. At fine-tuning scale, the consistency and correctness of each example matter more than the total count.

Why do AI developers worry about benchmark contamination?

Benchmarks are only meaningful if the model has not seen the test questions during training. When web-scale training data includes benchmark items, a model can score well by memorising answers rather than by reasoning. The GPT-3 authors found that some benchmarks were almost entirely present in their training data and excluded those results. Developers now try to keep evaluation data strictly separate from training data.

Does more training data always make a better model?

Not on its own. The Chinchilla research found that many large models were undertrained for their size: a smaller model trained on more tokens did better for the same compute. The same authors suggested that larger datasets are only beneficial when the data is high-quality. Published model papers describe extensive deduplication, filtering and quality scoring, and they sample sources they consider higher quality, such as Wikipedia, more often than raw web text.

Sources

• Language Models are Few-Shot Learners (GPT-3) — arXiv
• LLaMA: Open and Efficient Foundation Language Models — arXiv
• Training Compute-Optimal Large Language Models (Chinchilla) — arXiv
• Training language models to follow instructions with human feedback (InstructGPT) — arXiv
• LIMA: Less Is More for Alignment — arXiv
• The Llama 3 Herd of Models — arXiv
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — arXiv