Turning a Newspaper Archive into a Searchable Digital Collection

A newspaper archive becomes a searchable digital collection in four steps. First, scan each page, usually from microfilm, to a defined image standard. Second, run OCR that records where every word sits on the page. Third, package the results with issue, page and title metadata in a standard structure such as METS/ALTO. Many projects go further and segment each page into articles with headlines, bylines and dates, which makes article-level search, linking and tagging possible. The hard parts are specific to newsprint: dense multi-column layouts, small type on degraded paper, and articles that continue across pages.

This guide follows the path from microfilm or print to a working collection. It covers capture specifications, OCR problems in newsprint, article segmentation (often called zoning), headline, byline and date extraction, and the METS/ALTO package used by the US National Digital Newspaper Program. It then looks at linking and tagging articles and designing discovery interfaces. It ends with how four large public programmes have approached the job.

Key takeaways

• Many large newspaper programmes scan from microfilm, so the quality of the film, especially its generation and reduction ratio, sets a ceiling on OCR quality.
• Page-level OCR with word coordinates is enough for full-text search with hit highlighting, but article segmentation is what makes article-level search, tagging and AI use practical.
• METS/ALTO is the de facto packaging standard for digitised newspapers: ALTO holds text and coordinates for each page, and METS ties pages to issues, editions and titles.
• OCR error rates vary by decade, title and region of the page, and headlines can be worse than body text, so measure them by sample rather than assuming an average.
• Keep the master images and coordinates, because better OCR and layout models can be applied to the same pages years later.

From microfilm or print to page images

Newspapers are large, fragile and numerous. Many programmes therefore digitise from microfilm, which already exists for many historical runs and spares the originals further handling. The US National Digital Newspaper Program (NDNP) is a partnership between the National Endowment for the Humanities and the Library of Congress. Its technical guidelines for 2027–29 awards, dated 2 October 2026, specify:

• Source film: a second-generation (2N) duplicate silver negative made from the camera master. Titles on higher-quality microfilm are preferred.
• Reduction ratio: the lower the better, with below 20x preferred. If the ratio is too high to scan at 400 dpi, tests on sample images can decide whether 350 or 300 dpi still gives suitable OCR confidence.
• Capture: 8-bit greyscale at the maximum resolution possible between 300 and 400 dpi, relative to the original page.
• Processing: cropping to the page edge, and deskewing where text blocks are skewed by more than 3 degrees, before OCR. The master TIFF must have the same cropping and deskewing as the image used for OCR, but none of its other enhancements, such as bitonal conversion, sharpening or contrast adjustment.
• Targets: a standard scanning target film strip at the start of each session to monitor equipment performance.

When you scan from print instead, as many publishers do with their own bound volumes, the same principles apply. Use a published image standard, capture in greyscale or colour rather than pure black and white, and keep an unaltered master separate from the images you prepare for OCR.

OCR quality issues in newsprint

Newspapers combine almost every OCR difficulty at once: small type, narrow columns, cheap paper that yellows and breaks, ink show-through, column rules, adverts with decorative type, and, for some titles, Fraktur type or non-Latin scripts. A 2009 analysis for the National Library of Australia's newspaper programme found that raw, uncorrected OCR on a sample of 45 pages from 1803 to 1954 scored between 71% and 98.02%, measured by the engine's character confidence. It lists the main factors: the quality of the original and of the microfilm, scanning resolution, contrast and preprocessing, skew, complex layouts, and damage such as mould and fading.

More recent measurements show the same pattern. The American Stories project, a research pipeline applied to nearly 20 million Chronicling America scans, reported an end-to-end character error rate (CER) of 5.1% on a set of ten hand-transcribed pages. Headlines had a CER of 8.9% against 4.9% for article text. On a separate sample of lines from each decade, OCR CER ranged from 8.9% in the 1850s to 1.8% in the 1910s.

Practical implications:

• Measure by stratum. Sample by decade, title and region type (headline, body, advert), not just overall. Averages hide the worst collections.
• Record language per text block. NDNP requires the language of the text to be encoded per ALTO TextBlock using ISO 639-2 codes. Text with no language attribute is treated as English. Titles printed in Fraktur must use Fraktur-specific processing.
• Plan for reprocessing. In 2026 the Library of Congress released NDNP-Open-OCR, open-source software that reprocesses legacy newspaper OCR. It combines the Tesseract engine with layout models from American Stories. By August 2026 the Library reported reprocessing more than 425,000 pages, with work continuing on newspapers added before 2012. That is possible because the page images were kept alongside the original OCR.

Our guide to OCR for archives explains how to measure CER and WER and run quality-control sampling.

Article segmentation (zoning)

A newspaper page is a mosaic. Article segmentation, also called zoning or optical layout recognition, divides each page into regions and groups them into articles:

1. Detect regions: headlines, body text blocks, bylines, images, captions, adverts, tables, mastheads and page furniture.
2. Order the text within each article, column by column.
3. Associate regions into articles: link each headline to its text blocks, byline and illustrations.
4. Join continuations, where an article jumps to another page ("continued on page 6").

Not every programme does this. NDNP deliberately does not. Its approach is page-level OCR with word bounding boxes and column recognition, but without segmenting pages into articles. If an OCR process generates article zones, they must be removed before delivery. Other projects treat segmentation as core: the Europeana Newspapers project listed optical layout recognition (article segmentation) among its refinement methods, alongside OCR and named-entity recognition.

The American Stories pipeline shows what automated segmentation produces at scale:

Its article-association step is rule-based. It uses the position of headline, article and byline boxes to link headlines to the first article box below them, and reports an F1 score of 97.0 on a test set of headline–article pairs. The authors note that only around 3.8% of articles span multiple article boxes and 0.2% span pages. That release does not merge boxes that belong to the same article, so continuations still need separate handling.

Headlines, bylines, dates and METS/ALTO

Metadata fields

Once regions are classified, the key fields come from two places:

• From the issue: title, date, edition and page number. These are best captured once per issue during scanning, not inferred from OCR. NDNP identifies an issue by the title's LCCN plus issue date and edition order, and a page by adding its page sequence number. It also asks for records of issues and pages known to have been published but missing from the film, flagged as not present.
• From the page: headline, sub-headline, byline, dateline, section heading, caption and body text, taken from the classified regions. Normalise bylines, such as "By Our Own Correspondent" and agency credits, into separate fields. Parse datelines into place and date where possible.

The METS/ALTO package

The NDNP specification is a well-documented example of the package. For each page it requires:

• a master TIFF (8-bit greyscale)
• a JPEG 2000 derivative, used by the Library of Congress for zooming
• a searchable PDF with hidden text, its image downsampled to 150 dpi
• an ALTO XML file

The ALTO file must be UTF-8, one per page, with text in column-by-column reading order and bounding-box coordinates at word level. Confidence data and font information are optional extras. Pages, issues and reels are described in METS objects, with bibliographic data in MODS inside them.

A simplified ALTO fragment for one word looks like this:

<TextBlock ID="TB12" LANG="eng" HPOS="1210" VPOS="2034" WIDTH="980" HEIGHT="1450">
  <TextLine HPOS="1212" VPOS="2036" WIDTH="976" HEIGHT="38">
    <String CONTENT="Harbour" HPOS="1212" VPOS="2036" WIDTH="190" HEIGHT="38"/>
  </TextLine>
</TextBlock>

Those coordinates let an interface highlight search hits on the page image, the main way users check OCR text against the original. If you segment articles, keep the article structure in your own data model or METS structural map, linked to the ALTO blocks. Don't collapse everything into plain text.

Linking, tagging and discovery

Article-level text opens up several kinds of enrichment:

• Linking. Join continuations, link follow-up stories and corrections, and connect syndicated or reprinted copy that appears in many titles. The American Stories authors list detection of reproduced content and news story clustering among the uses of article-level text. Our guide to finding duplicates covers the techniques.
• Tagging. Label articles by topic, place, people, organisations and article type, ideally against a shared vocabulary such as IPTC Media Topics. Expect OCR errors to reduce entity-extraction accuracy, and measure it on samples.
• Crowd correction. The National Library of Australia opened OCR correction to the public in 2008. In the first 12 weeks, about 1,200 people corrected 700,000 lines in 50,000 articles.

For discovery interfaces, the basics are well established:

• browsing by title and date, which NDNP's page, issue, edition and title metadata is designed to support
• full-text search with hit highlighting on the page image
• article-level results where segmentation exists
• facets for date, place, title and topic
• downloadable images and text

Increasingly, users also expect APIs and bulk data. In 2026 the Library of Congress launched a Chronicling America datasets portal offering bulk OCR and batch data for more than 3,000 batches and 23 million pages. For newsrooms with their own archives, the same building blocks support related-story suggestions and archive search, as our guide to AI in the newsroom explains.

How four public programmes approached it

Two lessons stand out. First, choices made early, such as page-level versus article-level structure, shape every later use of the collection. Second, these collections are never finished. Trove's data guide notes that only around 6% of articles have user corrections. The Library of Congress is reprocessing OCR on pages digitised years ago. Plan your archive so it can be improved, not just published.

How Druid Learning helps

Druid Learning is an AI archival processing platform, based in Dublin and New York, that turns content archives into AI-ready datasets. Its Archive Optimization product turns archives in any format, including OCR scans, PDF and XML, into an indexed, labelled dataset. For NPR, Druid Learning processed more than 2 million assets with over 20 labels across eight dimensions, including article type, location and topic. That work supports suggested tags and related stories for journalists, archive search by label, and audience analytics by label. The Standardization Tool provides real-time content tagging and labelling.

Frequently asked questions

What is METS/ALTO in newspaper digitisation?

METS/ALTO is the de facto standard for packaging digitised newspapers, used by the US National Digital Newspaper Program and the basis of the Europeana Newspapers metadata model. ALTO is an XML format that stores the OCR text of a page with the coordinates of each block, line and word. METS is an XML wrapper that ties page files together into issues, editions and titles, and holds descriptive and technical metadata. Together they let systems search text and highlight hits on the page image.

What is article segmentation, and do I need it?

Article segmentation, or zoning, divides each page into regions such as headlines, text blocks, bylines, images and adverts, then groups them into articles. It isn't required for basic full-text search: the US National Digital Newspaper Program deliberately uses page-level OCR without it. It is needed for article-level search results, tagging, related-article links and most AI uses, because page-level text mixes many unrelated stories.

Should newspapers be digitised from microfilm or from print?

Microfilm is faster and protects fragile originals, and it is the main source for large programmes such as the US National Digital Newspaper Program. The trade-off is that film quality limits OCR quality. NDNP specifies second-generation duplicate negatives, prefers low reduction ratios and scans at 300 to 400 dpi relative to the original. Scanning from good print copies can give better images, but it means handling fragile originals page by page.

How accurate is OCR on historical newspapers?

It varies widely by period, title and page region. A 2009 Australian study found raw character confidence between 71% and 98.02% across 45 sample pages. A later research pipeline on Chronicling America reported an overall character error rate of about 5%, worse for headlines and for older decades. Measure your own collection on a stratified, hand-checked sample before deciding on correction or reprocessing.

Sources

• The National Digital Newspaper Program (NDNP) Technical Guidelines for Applicants, 2027–29 Awards — Library of Congress
• When Open Data Comes Full Circle: New OCR Reprocessing Software and Datasets Portal for Chronicling America — The Signal, Library of Congress
• American Stories: A Large-Scale Structured Text Dataset of Historical U.S. Newspapers — Dell et al., arXiv
• The British Newspaper Archive reaches 100 million pages — British Library
• Findmypast renews landmark partnership with the British Library — DC Thomson
• Europeana Newspapers: A Gateway to European Newspapers Online (project flyer) — Europeana Pro
• Understanding the digitised newspapers — Trove Data Guide
• How Good Can It Get? Analysing and Improving OCR Accuracy in Large Scale Historic Newspaper Digitisation Programs — D-Lib Magazine