What Makes Content Valuable for AI

AI developers value content that teaches their models something the open web cannot. That usually means original material at a useful scale, written or edited to a high standard and grounded in real expertise. It helps when the content covers many topics, languages and time periods, comes as clean text with clear structure and metadata, and carries a reliable record of where it came from. Duplicates, boilerplate and poorly scanned text lower its value, whatever the size of the collection.

These are not marketing claims. Over the past few years, research on training data has measured the effect of data quality directly. Filtering a web corpus for educational value raises benchmark scores. A small model trained largely on "textbook quality" data performs well on coding tasks despite its size. Deduplication reduces memorisation, and models trained on older text do worse on newer tasks.

This article sets out each quality, the evidence behind it, and how to check your own archive against it.

Key takeaways

Published research repeatedly finds that smaller amounts of high-quality, filtered data can match or beat much larger amounts of less carefully selected web text.
Content that is not already freely available online tends to add the most, because large web crawls are the common starting point for most developers.
Editorial quality, domain expertise, long-form depth and diversity of topics, languages and dates all have support in published research on training data.
Structure, metadata and clear provenance make content easier to filter, mix, evaluate and document, which raises its practical value.
Duplicates, templated boilerplate and OCR errors reduce usefulness, and duplicates and boilerplate are among the first things data pipelines remove.

Why quality now matters more than raw volume

Two lines of research explain why selectivity matters.

The first concerns supply. In a June 2024 analysis, Epoch AI estimated that the effective stock of public, human-generated text (counting only text good enough to train on, and allowing for training on the same data more than once) is on the order of 300 trillion tokens, with a 90% confidence interval of 100 to 1,000 trillion. Its 80% confidence interval is that, if current trends continue, this stock will be fully used at some point between 2026 and 2032. Epoch noted that AI-generated synthetic data had so far been shown to improve capabilities reliably only in relatively narrow domains, such as maths and coding.

The second concerns quality. When researchers built FineWeb, a 15-trillion-token dataset drawn from 96 Common Crawl snapshots, they also built FineWeb-Edu: 1.3 trillion tokens selected by a classifier trained on educational-quality scores that a large model, Llama-3-70B-Instruct, gave to sample web pages. In their larger test runs (a 1.7-billion-parameter model trained on 350 billion tokens), switching from FineWeb to FineWeb-Edu raised scores on the MMLU knowledge benchmark from 33% to 37%, and on the ARC reasoning benchmark from 46% to 57%. On MMLU, FineWeb-Edu matched the final score of another open web dataset, Matrix, with almost ten times fewer training tokens.

The "Textbooks Are All You Need" study went further. Its phi-1 model had only 1.3 billion parameters and was trained on 6 billion tokens of "textbook quality" web data plus 1 billion tokens of synthetic textbooks and exercises generated with GPT-3.5. It reached 50.6% pass@1 on the HumanEval coding benchmark, which the authors noted it achieved "despite this small scale".

The conclusion for content owners is straightforward. A developer with access to the whole web still wants more of the best material, and the best material is scarce.

The qualities AI developers look for

Quality	Why it matters	What the evidence shows	How to check your archive
Originality	Adds information models have not already seen	Public web text is the shared base; its supply is finite	What share is not freely available online?
Editorial quality	Clean, accurate, well-written text trains better models	Quality filtering improves results even though it removes data	Was it professionally edited or reviewed?
Domain expertise	Expert material is a minority of web text	Educational and textbook-style data raise benchmark scores	Is it written or checked by experts?
Long-form depth	Sustained explanation and argument	Mixing books with web text is broadly beneficial	How much is articles, chapters, reports?
Diversity	Broader coverage of topics, languages and periods	Heterogeneous sources help; data age affects performance	Range of subjects, languages and dates
Structure and metadata	Lets developers filter, mix and evaluate	Pipelines filter and weight data by its properties	Are dates, authors, subjects and sections recorded?
Provenance	Shows where each item came from	Many widely used datasets are poorly documented and hard to trace	Can each item be traced to a source and date?
Freshness	Keeps knowledge current	Models trained on older data do worse on newer tasks	How recent is the material? Is it updated?

Originality: material not already on the open web

Most large models start from the same raw material: crawls of the public web, filtered and deduplicated. Our guide to how AI models use data shows how large a share web data takes in published training mixes. Content that is already freely crawlable tends to add less that is new. Archives that were never published online, such as print-era newspapers, back catalogues, internal research or out-of-print books, often contain information that developers cannot easily get elsewhere.

Quality and editorial standards

Developers already filter for quality using automated signals. Meta's Llama 3 team experimented with several model-based quality classifiers, including fast classifiers trained to recognise whether a text "would be referenced by Wikipedia" and larger ones trained on quality judgements from their earlier Llama 2 model. Professionally edited content, such as journalism that has been fact-checked, textbooks that have been reviewed, or reports that have been copy-edited, is the kind of text these filters are designed to keep.

Domain expertise

FineWeb-Edu's classifier was explicitly trained to score educational value. At the threshold the researchers chose, it kept 1.3 trillion of FineWeb's 15 trillion tokens, under a tenth, and the model that produced its training labels was told to focus on grade-school and middle-school knowledge so that it would not favour highly technical pages. The phi-1 team selected its web code with a classifier trained on GPT-4 ratings of each sample's "educational value for a student whose goal is to learn basic coding concepts". Llama 3's developers built separate pipelines to find code and maths-related web pages, and used annealing, a short final training phase on a small amount of upsampled high-quality data, to judge the value of small domain-specific datasets. Curriculum-aligned educational material and expert writing in specific domains are the kinds of content these tools are built to pick out of the web.

Long-form depth

Long documents carry sustained reasoning, context and structure. A controlled study of pre-training data by Longpre and colleagues, which trained 28 models of 1.5 billion parameters, found that "the inclusion of heterogeneous data sources, like books and web, is broadly beneficial and warrants greater prioritization". Among the domains it measured, books had the longest and most readable documents. Books, long reports and in-depth features supply the kind of sustained text that short web pages rarely provide.

Diversity of topics, languages and time

Diversity works on several axes at once. Llama 3 reserved about 8% of its 15-trillion-token training mix for multilingual text. Time matters too. The same Longpre study found that a "temporal shift between evaluation data and pretraining data leads to performance degradation, which is not overcome by finetuning". An archive spanning decades offers historical depth, and one that keeps growing offers recency. Both are useful.

Structure and metadata

The pipelines in the research above all work by measuring properties of documents and acting on them: quality scores, line structure, domain, language, date. Content that arrives with headings, sections, captions, publication dates, authors and subject labels can be filtered and weighted precisely. The alternative is guesswork. Structure also makes content usable for retrieval, where an answer needs to cite a specific section.

Clear provenance

The Data Provenance Initiative traced the lineage of more than 1,800 text datasets used to fine-tune AI models. It found that widely used dataset collections are often "treated as monolithic, instead of a lineage of data sources", often after multiple rounds of repackaging by successive practitioners, and that efforts to attribute and document these ingredients are diminishing. The authors warned of "a crisis in misattribution and informed use" of the most popular datasets. Content that comes with a clear record of origin, date, author and editorial process is easier for developers to document and trust. Our guide to documenting a dataset covers how to write that record down.

Freshness

A model's knowledge stops at its training cut-off: Llama 3's pre-training data had knowledge until the end of 2023. Newer material is valuable both for training the next model and for retrieval systems that supply current information at answer time.

What lowers usefulness

Duplicates and near-duplicates

Duplication is a widespread problem in large collections. Lee and colleagues found a single 61-word English sentence repeated more than 60,000 times in the C4 web dataset. They also found that over 1% of the unprompted output of models trained on such data was copied verbatim from the training set. After deduplication, models emitted memorised text ten times less often and needed fewer training steps to reach the same or better accuracy. Deduplication also reduced overlap between training and test data, which affected more than 4% of the validation sets of standard datasets, making evaluation more accurate.

In archives, duplicates come from syndicated wire copy, multiple editions of the same story, reprints and repeated scans. See finding duplicates and near-duplicates.

Boilerplate and templated text

Repeated lines add little new information. Llama 3's line-level deduplication removed lines that appeared more than six times in each group of 30 million documents. The authors noted that this removed leftover boilerplate such as navigation menus and cookie warnings, and also some frequent high-quality text, but their evaluations still showed strong improvements. FineWeb's custom filters target similar patterns, removing documents with too many characters in duplicated lines or too many very short lines. In publishing archives, the equivalents are mastheads, page furniture, standing headers, standard footers and repeated subscription prompts.

Poor OCR and conversion errors

Errors introduced by scanning carry through to everything built on the text. A 2020 study by van Strien and colleagues tested sentence segmentation, named entity recognition, dependency parsing, information retrieval, topic modelling and language model fine-tuning on OCR output. They found "a consistent impact resulting from OCR errors" across these tasks, with some tasks "more irredeemably harmed" than others. Our guide to OCR for archives covers how to measure and improve accuracy.

Missing context

Text without dates, authors or document boundaries is harder to filter, deduplicate, evaluate and attribute. A pile of extracted text with no metadata is worth less than the same text with it.

A self-assessment checklist

Before discussing an archive with an AI developer, work through these questions:

Scope. How many documents, roughly how many words, which formats, which date range?
Exclusivity. What share is not already freely available on the open web?
Quality. Was the content edited, reviewed or fact-checked? Is there a style or editorial standard?
Expertise. Which subjects does it cover in depth, and who wrote it?
Diversity. Which languages, regions, topics and decades are represented, and where are the gaps?
Duplication. What share is duplicated or near-duplicated, for example through syndication, editions or repeated scans?
Text quality. For scanned material, what is the measured OCR accuracy on a sample?
Metadata. Are dates, authors, titles, sections and subjects recorded consistently?
Provenance. Can each item be traced to its original source and date?
Freshness. Is the archive still growing, and can new material be delivered regularly?

A generic example: a regional newspaper's archive covering 1950 to 2020 might contain millions of articles. Much of the print-era content was never online, which is a strength for originality and historical depth. But it may contain heavy duplication from wire copy, uneven OCR on older pages, and incomplete metadata before the paper adopted a digital system. Fixing those three issues often matters more to its value than adding volume.

How Druid Learning helps

Druid Learning is an AI archival processing platform, based in Dublin and New York, that turns content archives into AI-ready datasets. Archive Optimization processes archives in any format, including OCR scans, PDF, XML, IDML, INDD, EPUB, WAV, MP3 and MP4, into an indexed, labelled dataset. For NPR, Druid processed 2M+ assets with 20+ labels across 8 dimensions (case study). Through Data Partnerships, Druid prepares an archive, raw or structured, and connects content owners with AI companies.

Frequently asked questions

Is content that is already online still valuable to AI developers?

It can be, but it usually adds less. Large web crawls are the common starting point for training data, so freely available pages have often already been collected and filtered. Online content gains value when it is delivered in a cleaner, better-structured form than a crawl provides, with reliable dates and metadata, or when it is current material needed for AI answers at the time a question is asked.

Does old archive content have value for AI?

Yes. Research on pre-training data found that a mismatch between the time period of training data and the time period of a task degrades performance, and that fine-tuning does not overcome it. Archives spanning decades offer historical language, events and context that recent web text lacks. Their value depends heavily on text quality, so OCR accuracy and metadata such as publication dates matter a great deal.

How much does OCR quality matter for AI use?

It matters significantly. A 2020 study testing six common language-processing tasks on OCR output, including named entity recognition, information retrieval and language model fine-tuning, found a consistent negative impact from OCR errors, with some tasks harmed more severely than others. For scanned archives, measuring accuracy on a sample and correcting the worst material is usually worthwhile before the text is used for AI.

Why do duplicates matter so much in large archives?

Duplicates waste training effort and encourage memorisation. Research on web training data found a single sentence repeated more than 60,000 times in one widely used dataset, and showed that removing duplicates made models emit memorised text ten times less often. In content archives, duplicates come from syndicated copy, multiple editions, reprints and repeated scans. Finding and resolving them is usually one of the first steps in preparing an archive.

What Makes Content Valuable to AI Developers?