To digitise a content archive for AI, take four steps. First, inventory what you hold. Next, capture it to a published imaging standard, such as FADGI or Metamorfoze, so you never have to rescan. Then extract the text and structure, with OCR for scans and direct extraction for born-digital files. Finally, deliver clean, well-described records with stable identifiers that an AI team can load, filter and trace back to the original. The scanning is usually the easy part. The work that decides whether the result is useful is in planning, metadata and quality control.

This guide walks through a digitisation project end to end. It covers auditing and prioritising collections, choosing scanning resolution and file formats, OCR and text extraction, structure and metadata, quality control, and storage. It finishes with how to package the output for AI use and a project-planning checklist. The advice applies whether the archive is a newspaper morgue, a publisher's back catalogue, an educational resource library or a research collection.

Key takeaways

  • Start with an inventory and a clear statement of intended uses, because uses drive every later decision on resolution, formats and metadata.
  • Capture once to a recognised standard: FADGI recommends at least its three-star level where possible, which for most text-based materials means a nominal 300 ppi.
  • Keep unaltered master files separate from the derivatives you OCR, publish or feed to models, and record how each derivative was made.
  • Measure OCR and metadata quality on samples instead of trusting engine confidence scores, which correlate poorly with real accuracy.
  • An AI-ready output is text plus structure plus provenance: every record should trace back to a specific master file and page.

Audit, inventory and prioritise

Before you scan anything, find out what exists. An inventory doesn't need to describe every item. It needs enough detail to size the job, spot problems early and make priority calls. The Digital Preservation Coalition's handbook recommends checking whether the material has already been digitised to a usable standard. It also advises budgeting recurrent funds for maintenance on top of one-off conversion costs.

For each series or collection, record at least:

Field Why it matters
Collection name and date range Defines batches and sets expectations for typefaces, languages and layout
Physical or digital format Bound volumes, loose sheets, microfilm, born-digital files, audio and video each need different workflows
Approximate volume Pages, files or hours, which drive cost and schedule
Condition Fragile, brittle or damaged items need slower handling and specialist equipment
Existing digital copies Earlier scans or exports may be reusable, or may need redoing
Existing metadata Catalogue records, spreadsheets and file-naming schemes you can carry forward
Rights and restrictions status As recorded by your organisation, so restricted material can be flagged in the output

Then set priorities. A practical scoring approach weighs four things for each collection. Each is a judgement call that belongs to your organisation:

  1. Value of use: how much demand there is from editors, researchers, learners or AI partners.
  2. Risk: the condition of the originals and the obsolescence of their carriers.
  3. Uniqueness: whether this is the only copy anywhere.
  4. Cost per unit: clean modern print is cheap per page, while fragile bound volumes and complex layouts are expensive.

Start with a pilot of a few thousand pages drawn from across your collections, deliberately including the hardest material rather than only the easiest. The pilot shows real throughput, OCR quality and metadata effort before you commit to a full budget.

Capture: resolution and file standards

Two published guidelines are widely used in cultural heritage imaging. Using one of them gives you a defensible quality target and a way to measure it.

FADGI star levels

The US Federal Agencies Digital Guidelines Initiative published the third edition of its Technical Guidelines for Digitizing Cultural Heritage Materials in May 2023. It defines four performance levels, from one star to four stars. These correspond to ISO 19264 Levels C, B and A at two, three and four stars, with one star offered for reference use only. FADGI describes two stars as the minimum acceptable for most professional work and three stars as a very good professional image suitable for most uses. To avoid future rescanning, it recommends not digitising below three stars where possible. It also cautions against four-star imaging for all but the most sophisticated programmes.

Nominal resolutions for some common material types in the 2023 edition:

Material category (FADGI 2023) 2-star 3-star 4-star
Bound volumes: general collections 250 ppi 300 ppi 400 ppi
Bound volumes: rare and special 250 ppi 300 ppi 400 ppi
Unbound documents: general collections 250 ppi 300 ppi 400 ppi
Newspapers 250 ppi 300 ppi 400 ppi
Microfilm (relative to frame size) 3,500 ppi 4,000 ppi 4,500 ppi

The figures for paper originals carry a small tolerance, for example at least 294 ppi for a nominal 300 ppi, while the microfilm figures are minimums measured against the film frame. Resolution is only one parameter. FADGI also measures tone response, white balance, colour accuracy, uniformity and sharpness against imaging targets. For newspapers it recommends three-star performance, which it reports was tested with OCR and gave excellent results across languages and paper conditions. Master formats listed across most categories are TIFF, JPEG 2000 and PDF/A.

Metamorfoze

The Dutch Metamorfoze programme, hosted by the KB National Library, published version 2.0 of its Preservation Imaging Guidelines in April 2025, replacing the 2012 version. It has three levels:

  • Metamorfoze Full is for originals treated as works of art, such as illuminated manuscripts, maps and photographs.
  • Metamorfoze Light is for books, newspapers, magazines and manuscripts where colour accuracy matters somewhat less.
  • Metamorfoze Extra Light is for text documents up to A3 on sheet-fed scanners, where only textual information is guaranteed.

Full and Light specify 300 ppi for originals of A5 and larger, and allow smaller originals to be digitised at up to 600 ppi. The guidelines assume preservation masters are primarily uncompressed TIFF 6.0.

Masters and derivatives

Whichever standard you follow, keep the master as an unaltered first-generation capture. Do cropping, deskewing, contrast enhancement and compression on derivatives made from it. Metamorfoze is explicit that preservation masters should not be stitched or focus-stacked, and that such edits belong in user files. Record the scanner, settings and software used, because the DPC handbook advises recording the digitisation method, compression, bit depth and resolution in the metadata.

Text extraction and OCR

AI systems consume text, so this step matters most to the people who will use your dataset.

Born-digital files (PDF, XML, IDML, EPUB, word-processing files) usually already contain text. Extract it directly rather than rendering the pages and running OCR on them. Each format hides its own traps, such as reading order in PDFs and text threaded across frames in InDesign files. See our guide to publishing file formats.

Scanned material needs optical character recognition. Practical points:

  • Feed the OCR engine a derivative prepared for it: deskewed, cropped and, if needed, contrast-adjusted. The open-source Tesseract engine's documentation says it works best on images of at least 300 dpi.
  • Choose output with coordinates, such as ALTO XML or hOCR, not plain text. Word positions let you highlight search hits on the page image and re-derive text later.
  • Set the language or languages explicitly, and use models suited to historical typefaces where you need them.

OCR quality on historical material varies widely. In a 2009 study for Australia's newspaper digitisation programme, raw, uncorrected OCR on a 45-page sample from 1803 to 1954 scored between 71% and 98.02%, measured by the engine's character confidence. Errors carry through to downstream tasks: a 2021 survey of OCR evaluation tools reports a 2019 study that added OCR errors to a named-entity dataset, in which recognition accuracy fell from 90% to 60% as the error rates rose. Our guide to OCR for archives covers measurement and correction in detail.

Structure, metadata and quality control

Plain text strips away information that both people and models need. Capture three kinds of metadata alongside it:

  • Descriptive: title, creator, date, publication, section, subject, language.
  • Structural: page order, issue or volume, article or chapter boundaries, headings, captions, tables.
  • Administrative and technical: source file, capture device and settings, OCR engine and version, checksums, rights status, processing date.

Use controlled values wherever possible: ISO 8601 dates (the DPC recommends YYYY-MM-DD, including in file names), standard language codes and an agreed subject vocabulary. Free-text fields that differ from batch to batch are a common reason archive metadata can't be filtered reliably.

Quality control should be designed in, not bolted on at the end:

  1. Image QC: analyse imaging targets at regular intervals against your chosen standard. Both FADGI and Metamorfoze rely on measuring technical targets, and Metamorfoze also requires visual checks for artefacts.
  2. Completeness: compare page and file counts against the inventory and check that sequences have no gaps or duplicates.
  3. OCR QC: transcribe a random sample of pages or regions by hand and calculate character and word error rates. Engine confidence scores are not a substitute. In the 2021 survey's experiments, confidence had at best a slight correlation with measured character error rates.
  4. Metadata QC: spot-check a sample of records against the originals, and run automated checks for missing, malformed or out-of-range values.

Set acceptance thresholds before production starts and record the results with each batch.

Storage and preservation

Digitised files are only useful while they survive intact. The essentials:

  • Fixity: generate a checksum for every file at creation and verify it on every transfer. The DPC's guidance says MD5 is sufficient for detecting accidental damage and quick to calculate, while stronger algorithms such as SHA-256 make deliberate tampering harder to hide. It suggests rechecking stored content periodically, as a guideline roughly annually for tape and every six months for disk-based systems.
  • Multiple copies: checksums only tell you something changed. Repair depends on having another good copy, ideally in a separate location.
  • Format choice: the Library of Congress Recommended Formats Statement lists XML with a documented schema, EPUB 3, PDF/A and PDF/UA among its preferred formats for digital textual works. For page images, the TIFF and JPEG 2000 masters recommended by FADGI are widely supported.
  • Backups at the right moments: the DPC advises making backups immediately after conversion and again when loading data into systems.

Preparing outputs for AI

An AI team will want the content in a form it can load, filter, deduplicate and trace back to its source. In practice that means:

  • One record per meaningful unit: an article, chapter, lesson, page or clip, depending on the collection. Don't deliver one giant file per volume.
  • Stable identifiers that link every record to its master file and page.
  • Clean text with headers, footers and page furniture separated from body text, and hyphenation at line ends resolved.
  • Metadata as fields, not embedded in the text.
  • Quality signals, such as measured OCR error rates per batch, so users can filter.
  • Documentation of what the dataset contains, how it was made and its known gaps.

A single record in a JSON Lines (JSONL) delivery might look like this:

{"id": "gazette-1923-05-14-p03-a02", "master_file": "masters/1923/05/14/0003.tif", "page": 3, "date": "1923-05-14", "language": "en", "section": "Local news", "title": "New bridge opens to traffic", "text": "The new bridge over the river was opened...", "ocr_engine": "engine-name 5.x", "batch_cer": 0.031, "sha256": "9f2c..."}

Our guides to dataset delivery formats and documenting a dataset cover the next stage.

A project-planning checklist

  • Intended uses written down and agreed with stakeholders
  • Inventory complete, with volumes, formats, condition and existing metadata
  • Priorities scored, and a pilot batch that includes difficult material
  • Imaging standard and target level chosen (for example FADGI three-star), with QC frequency set
  • Master and derivative formats specified, with a file-naming scheme
  • OCR engine, output format (ALTO or hOCR), languages and models chosen
  • Metadata schema and controlled vocabularies agreed
  • Sampling plan and acceptance thresholds for image, OCR and metadata QC
  • Checksums generated at creation and verified on every transfer
  • Storage with multiple copies and a periodic fixity check schedule
  • AI output specification: record unit, identifiers, fields, delivery format and documentation
  • Budget that covers ongoing storage and maintenance, not just conversion

How Druid Learning helps

Druid Learning is an AI archival processing platform, based in Dublin and New York, that turns content archives into AI-ready datasets. Archive Optimization takes archives in any format, including OCR scans, PDF, XML, IDML, INDD, SVG, EPUB, WAV, MP3 and MP4, and turns them into an indexed, labelled dataset. The platform has processed more than 10 million assets and handles content in over 140 languages. For example, it processed 50,000 educational resources for the Irish publisher CJ Fallon and mapped them to the curriculum. Through Data Partnerships, it prepares an archive, raw or structured, and connects content owners with AI companies.

Frequently asked questions

What resolution should I scan documents at for OCR and AI use?

For most printed text, a nominal 300 ppi is the common baseline. It matches FADGI's three-star level for bound volumes and newspapers, and Metamorfoze's requirement for originals of A5 and larger. Tesseract's documentation also says it works best at 300 dpi or more. Small type, small originals and rare material may justify 400 ppi or more. Test OCR quality on a pilot batch before you fix the setting.

Should I keep TIFF masters if I already have searchable PDFs?

Yes, if you can. A searchable PDF is usually a compressed derivative with an OCR text layer. A TIFF or JPEG 2000 master keeps the full captured image, so you can rerun better OCR later, create new derivatives and check output against the source. FADGI lists TIFF, JPEG 2000 and PDF/A as master formats. What matters is keeping one unaltered, well-documented master per page.

How do I know whether my OCR is good enough?

Measure it. Take a random sample of pages or regions, have people transcribe them accurately, and calculate the character error rate and word error rate against the OCR output. Don't rely on the confidence scores that OCR engines report: a 2021 survey of OCR evaluation tools found they had at best a slight correlation with measured error. Set a threshold for each collection based on how the text will be used.

What makes a digitised archive AI-ready rather than just online?

An online archive is built for people browsing page images. An AI-ready archive delivers clean text in meaningful units, such as articles or chapters, with structured metadata fields and stable identifiers that link back to the source page. It also carries measured quality indicators and documentation of how it was produced. That lets AI teams filter, deduplicate, evaluate and trace every record.

Sources