Data Quality for AI: Measuring and Improving a Content Dataset

Data quality for an AI dataset is the degree to which its content and metadata are fit for the use an AI team has in mind: whether the text is accurate, the records complete, the values consistent and valid, the information current, and nothing counted twice. You measure it by turning each of those dimensions into a rule you can count, checking samples by hand where rules cannot reach, and reporting the results together with their uncertainty.

Most content owners know their archive has problems: a gap in the 1970s, OCR that falls apart on one typeface, a metadata field that quietly changed meaning after a system migration. Few have numbers, and numbers are what a data partner, or your own AI team, needs to decide whether a dataset suits a task and how much work it requires.

This guide covers the recognised dimensions of data quality, how to measure each one for articles, books or transcripts, how to design spot-checks and read confidence scores, and what a useful quality report contains.

Key takeaways

• Data quality means fitness for a stated use, so define the use and the rules that follow from it before measuring anything.
• DAMA UK's six dimensions (completeness, uniqueness, timeliness, validity, accuracy and consistency) give a practical vocabulary, and ISO/IEC 25012 defines a broader model of fifteen characteristics.
• Validity, completeness, consistency and uniqueness can usually be checked automatically on every record, while accuracy normally needs a person comparing the data with its source.
• A random, stratified sample of a few hundred items gives a usable estimate of an error rate, provided the margin of error is reported alongside it.
• Model confidence scores become useful only after you have checked how well they match observed accuracy on your own content.

Two recognised frameworks

DAMA UK's six dimensions

In 2013 a DAMA UK working group published The Six Primary Dimensions for Data Quality Assessment, written to give practitioners and business stakeholders a shared vocabulary. It defines a data quality dimension as a recognised term for a feature of data that can be measured or assessed against defined standards. The UK government's Data Quality Framework (2020) uses the same six dimensions and adapts the paper's worked examples. The six are:

• Completeness: the proportion of stored data against the potential of 100% complete.
• Uniqueness: nothing is recorded more than once, based on how that thing is identified. The paper notes that uniqueness is the inverse of the level of duplication.
• Timeliness: the degree to which data represents reality from the required point in time.
• Validity: data conforms to the syntax (format, type and range) of its definition.
• Accuracy: the degree to which data correctly describes the real-world object or event.
• Consistency: the absence of difference when two or more representations of a thing are compared against a definition.

The paper is explicit that this is a checklist rather than a prescription: organisations choose the dimensions that matter, weight them, and set thresholds for good and bad quality. It also makes a point that is easy to forget: most dimensions can be assessed by analysing the data itself, but accuracy can only be assessed against the real thing the data represents, or against an authoritative reference.

ISO/IEC 25012

ISO/IEC 25012, part of the ISO/IEC 25000 family of standards, defines a data quality model of fifteen characteristics, each framed "in a specific context of use". They are grouped by whether quality depends on the data itself, on the system holding it, or on both:

Two refinements are especially useful for content. ISO distinguishes syntactic accuracy (closeness to values that are syntactically correct) from semantic accuracy (closeness to the right value), a line similar to DAMA's split between validity and accuracy. And traceability, an audit trail of access and changes to the data, is close to what AI teams call provenance.

Measuring each dimension in a document dataset

Both frameworks translate directly to articles, books and transcripts.

A few notes on applying these:

• Accuracy is usually the hardest dimension to measure. For text, have a person transcribe or correct a sample of pages carefully, then compare the dataset with that reference. For metadata, compare sampled records with the printed source: is the date the one on the masthead, is the byline right? Our guide to OCR for archives covers text accuracy in more depth.
• Completeness works at two levels. Field completeness is a simple count; DAMA's worked example is 294 of 300 records populated, or 98%. Coverage, meaning whether every issue, book or episode is present at all, needs an external list such as a publication schedule or catalogue. DAMA advises measuring critical fields first, because gaps in non-critical fields may not matter.
• Consistency is best found by listing every distinct value in a field with its count. 'Sport', 'Sports' and 'SPORT' will appear side by side, and so will dates written three different ways.
• Validity is not accuracy. DAMA's accuracy example is a date of birth entered in US month-first format on a European school application: valid in its US context, wrong in meaning. A record can pass every schema check and still be wrong.
• Uniqueness has its own methods, covered in our guide to finding duplicates and near-duplicates.

Even a quick look can expose serious problems. In Quality at a Glance, Julia Kreutzer and colleagues manually audited 205 language-specific corpora drawn from five major public web-crawled datasets. At least 15 contained no usable text, a significant fraction had fewer than 50% of sentences of acceptable quality, and many were mislabelled or used ambiguous language codes. Lower-resource corpora had systematic problems, and the authors show that these were easy to detect even for people not proficient in the language.

Automated checks and human spot-checks

Run automated rules over every record for validity, completeness, consistency and uniqueness: they are cheap, repeatable and free of sampling error. Reserve human effort for accuracy, and for checking that the automated rules themselves measure what you think they do.

Designing a sample

• Draw it at random. "The first hundred files" is often the most recent, cleanest or most-handled material, and it will flatter the results.
• Stratify by what drives quality. Error rates vary systematically by decade, title, typeface, source format and language, as the Kreutzer audit shows for languages. Sample within each group that matters and report the groups separately.
• Fix the sample before looking. Choose the item IDs first and record them, so nobody is tempted to swap in a better example.
• Record structured judgements. Capture pass or fail for each field and an error type, not just an overall impression, so results can be recounted and compared over time.
• Double-check a subset. Have two reviewers judge some of the same items to see whether they agree. Our guide to evaluating AI tagging covers agreement and review in more detail.

How many items to check

The most frequently used confidence interval for a proportion, as the NIST/SEMATECH e-Handbook of Statistical Methods describes it, is p ± z × √(p(1 − p) / n), where p is the observed error rate, n the sample size and z about 1.96 for 95% confidence. Rearranged for a target margin E, the sample size is n = z² × p(1 − p) / E². The margin is widest when p is 0.5, which gives two useful rules of thumb:

• About 385 randomly sampled records give a margin of error of at most five percentage points.
• About 1,068 records give at most three percentage points.

As a worked example, if 20 of 400 sampled articles have a wrong publication date, the estimate is 5% with a margin of about 2.1 points, so roughly 3% to 7%. When errors are rare, this simple formula can produce an impossible negative lower limit; the NIST handbook prefers the Wilson method, one advantage being that its lower limit cannot be negative. Each stratum you report separately needs its own sample of that size, so choose a few strata that genuinely matter.

Confidence scores: useful once calibrated

Many fields are produced by models (OCR, transcription, classification, metadata extraction), and most output a confidence score. It is tempting to treat a score of 0.9 as meaning "90% likely to be right". That holds only if the model is well calibrated. Chuan Guo and colleagues found that modern neural networks are often poorly calibrated, and that a simple post-processing method called temperature scaling was surprisingly effective at correcting this on most of the datasets they tested.

You do not need to retrain anything. Check the scores on your own content:

1. Group predictions into score bands, for example 0.5–0.6, 0.6–0.7 and so on up to 0.9–1.0.
2. Sample items from each band and check them by hand.
3. Compare the observed accuracy in each band with its scores.
4. Set thresholds per field: accept above one level, send the middle band for review, and leave the field empty below the lower level.
5. Deliver the scores with the data and state in the documentation how they were checked.

Scores from different models, or for different fields from the same model, are rarely comparable, so calibrate each one separately.

Writing a quality report

DAMA UK describes a typical assessment cycle: identify the critical data items, choose dimensions and weightings, define what good and bad look like, apply the checks, review the results, correct the data and the processes that produced the errors, and repeat periodically to track trends. A quality report records one turn of that cycle. A useful one includes:

• Scope: the dataset version, date and record counts.
• Purpose: the intended use and the rules derived from it.
• Results by dimension: with numerators and denominators, not only percentages.
• Sampling: the design, size and margin of error for every sampled measure.
• Breakdowns: results for each stratum that matters, such as decade, title or language.
• Known issues: described with concrete examples.
• Changes: what improved or worsened since the previous report.

An illustrative excerpt for a fictional newspaper dataset might look like this:

The report belongs with the dataset's wider documentation; our guide to documenting a dataset explains where it fits.

Common mistakes

• Collapsing everything into one score. A single quality figure hides which dimension failed. DAMA notes that each dimension is likely to carry a different weighting.
• Measuring what is easy rather than what matters. Validity is simple to automate; accuracy is what users notice.
• Fixing only the delivered copy. Correct the process that produced the error, then regenerate, or the same problem returns in the next release.
• Treating quality as a one-off. Data, tools and requirements change, so repeat the assessment for each release.

How Druid Learning helps

Druid Learning is an AI archival processing platform that turns content archives into AI-ready datasets. Archive Optimization produces an indexed, labelled dataset from archives in formats including OCR scans, PDF, XML, EPUB and audio and video files. For the Netherlands Cancer Institute, the work covered 1,000+ genomics studies from NCBI GEO, linked to 660 published papers: more than 20,000 research samples were enriched with standardised metadata, producing over 100,000 structured data points, each with a confidence score.

Frequently asked questions

What are the main dimensions of data quality?

A widely used set comes from DAMA UK's 2013 white paper: completeness, uniqueness, timeliness, validity, accuracy and consistency. ISO/IEC 25012 defines a broader model of fifteen characteristics, adding items such as credibility, traceability, precision and understandability. For a content dataset, the DAMA six cover most practical checks, and traceability is close to what AI teams call provenance.

How do you measure the accuracy of OCR or transcribed text?

Take a random sample of pages or recordings, have a person produce a careful reference transcription, and compare the dataset's text with it, counting character or word errors. Stratify the sample by period, source and format, because quality varies systematically, and report the error rate for each group with a margin of error rather than one headline figure for the whole archive.

How many records should we check by hand?

For a single error rate at 95% confidence, a simple random sample of about 385 records gives a margin of error of at most five percentage points, and about 1,068 records gives at most three points. If you need separate estimates for each decade, title or language, you need a sample of that size in each group, so choose the few groups that matter most.

Are AI confidence scores reliable?

Only once they have been checked. Research by Guo and colleagues found that modern neural networks are often poorly calibrated, meaning their stated confidence does not match how often they are right. Group predictions by score, check a sample from each group by hand, and compare observed accuracy with the scores before using them to accept, review or discard machine-generated values.

What should a data quality report include?

It should state the dataset version and intended use, the rules derived from that use, and results for each dimension with counts as well as percentages. For sampled measures, include the sample design, size and margin of error. Add breakdowns by meaningful groups, known issues with examples, and changes since the previous report, so readers can see whether quality is improving.

Sources

• The Six Primary Dimensions for Data Quality Assessment (October 2013) — DAMA UK Working Group
• The Government Data Quality Framework — UK Government Data Quality Hub, GOV.UK
• ISO/IEC 25012 data quality model — ISO 25000 portal
• Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets — Kreutzer et al., arXiv
• On Calibration of Modern Neural Networks — Guo et al., arXiv
• Confidence intervals for a proportion — NIST/SEMATECH e-Handbook of Statistical Methods