Product

Archive Optimization Standardization Tool Suggestion Engine Enhanced Analytics Data Partnerships Case studies Articles Press

Articles

Guides to AI-ready data, archives and AI

27 in-depth guides for publishers, newsrooms, educators and researchers: how to digitise and structure an archive, describe it with good metadata, make it searchable with AI, and work with AI companies.

AI-ready data

Data Quality for AI: Measuring and Improving a Content Dataset

Learn how to measure the quality of a document dataset for AI with DAMA's six dimensions, ISO/IEC 25012, spot-check sampling, confidence scores and reports.

Delivering a Dataset: JSONL, Parquet and Other Formats AI Teams Use

Learn when to deliver a dataset as JSON Lines, CSV, Parquet or WebDataset, how to design a document record, and how to size, compress and checksum the files.

Documenting a Dataset: Datasheets, Data Cards and Nutrition Labels

Learn how datasheets, data cards, nutrition labels and dataset cards document a dataset, what each section should cover, and a template for a content archive.

Finding Duplicates and Near-Duplicates in Large Content Archives

Learn why archives fill with duplicates, how hashing, MinHash, LSH and SimHash find them, why it matters for AI training data, and how to set thresholds.

What Is AI‑Ready Data? A Guide for Content Owners

Learn what makes a content archive AI‑ready: machine-readable text, clear structure, consistent metadata, provenance, deduplication, documentation and format.

Archives & digitisation

How to Digitise a Content Archive for AI: A Step-by-Step Guide

A step-by-step guide to digitising an archive for AI use: inventory, priorities, scanning standards, OCR, metadata, quality control, storage and AI outputs.

Making Audio and Video Archives Searchable

How to make audio and video archives searchable: speech-to-text, word error rate, diarisation, timestamps, SRT and WebVTT, shot detection and AV metadata.

OCR for Archives: Accuracy, Historical Documents and Quality Control

How OCR works, why historical documents are hard to read, how to measure accuracy with CER and WER, which engines and output formats to use, and how to run QC.

PDF, XML, IDML, InDesign and EPUB: What Publishing Files Contain

What PDF, PDF/A, XML (JATS, NewsML-G2), IDML, INDD, EPUB, SVG and HTML files hold, how to extract text, structure and metadata, and the pitfalls of each.

Turning a Newspaper Archive into a Searchable Digital Collection

How newspaper archives go from microfilm or print to searchable collections: scanning, OCR, article segmentation, METS/ALTO, metadata, linking and discovery.

Metadata & taxonomy

Evaluating AI Tagging: Precision, Recall and Human Review

Learn to measure AI tagging with precision, recall and F1, micro and macro averages, gold-standard samples, Cohen's kappa, thresholds and drift checks.

How to Build a Content Taxonomy: A Step-by-Step Guide

Learn how to build a content taxonomy step by step, from purpose and content audit to structure, synonyms in SKOS, governance, testing and AI tagging.

IPTC Media Topics and News Taxonomies Explained

Learn what IPTC Media Topics is, how its 17 top-level topics and multilingual labels work, how newsrooms extend it, and how to map legacy tags to it.

Metadata for Publishers: Descriptive, Structural and Administrative

Learn the main types of metadata, the standards publishers and newsrooms use, what good metadata makes possible, and how to fix the most common problems.

AI & search

Chunking Long Documents for AI Retrieval: Strategies and Trade-offs

Learn how to split long documents for AI retrieval, comparing fixed-size, structure-aware and semantic chunking, overlap, chunk size, metadata and PDFs.

Embeddings and Vector Search, Explained Without the Maths

Learn in plain language how embeddings, cosine similarity, vector databases, HNSW and hybrid search work, and when keyword search is the better choice.

Multilingual Archives: Language Detection, Translation and AI

Learn how to detect languages, normalise Unicode text, run OCR in other scripts, choose translation or multilingual models and search across languages.

Retrieval-Augmented Generation (RAG) Explained for Content Teams

Learn how retrieval-augmented generation works, why it reduces hallucination, what content owners must get right and how to evaluate a RAG system.

Data partnerships

AI Crawlers and robots.txt: A Practical Guide for Publishers

Learn how AI crawlers such as GPTBot, ClaudeBot and PerplexityBot identify themselves, how to address each in robots.txt, and where its limits lie.

How AI Models Use Data: Pre-training, Fine-tuning and Evaluation

Learn how AI models use data at each stage, from pre-training and fine-tuning to preference data, evaluation and retrieval, and why data quality matters.

Publishers and AI Companies: A Timeline of Content Partnerships

This dated timeline covers publicly announced content partnerships between publishers and AI companies from 2023 to 2026, and the patterns behind them.

What Makes Content Valuable to AI Developers?

Learn which qualities AI developers look for in content, from originality and editorial quality to structure and provenance, and what reduces its value.

Why Human-Created Content Matters: Synthetic Data and Model Collapse

Learn what synthetic data is, what model collapse research found, why keeping real data helps avoid it, and why human-created content keeps its value.

Sector guides

AI in Educational Publishing: Curriculum Alignment and Resources

How educational publishers use AI to map resources to curriculum outcomes, apply LRMI and LOM metadata, reuse content, improve accessibility and keep review.

AI in the Newsroom: Tagging, Related Stories and Archive Search

Learn where AI helps newsrooms beyond writing stories: tagging, entity extraction, related stories, archive search, transcription and editorial safeguards.

Content Analytics Beyond Pageviews: Labels and User Needs

Why pageviews mislead, how to label content by topic, format and user need, and how to send those labels to GA4 and Chartbeat to guide editorial decisions.

Research Data and AI: FAIR Principles and Reusable Datasets

What the FAIR principles mean for research data, why free-text metadata in repositories like NCBI GEO is hard to reuse, and how AI with human checks can help.

Want help with your own archive?

Talk to us about turning your content into an AI-ready dataset.