Evaluating AI Tagging: Precision, Recall and Human Review

You evaluate AI tagging by comparing the labels a model assigns with a trusted, human-reviewed sample, called a gold standard, and measuring two things: precision, the share of assigned tags that are correct, and recall, the share of correct tags the model found. Their harmonic mean, the F1 score, summarises both in one number.

With many labels, those numbers must be calculated per label and averaged without hiding weak ones. You also need to know how well your annotators agree, where to set confidence thresholds, which tags people should review and how to notice when quality slips. This guide covers each step with worked examples and ends with an evaluation plan.

Key takeaways

• Precision is the share of applied tags that are correct, recall is the share of correct tags the model found, and F1 balances the two; accuracy is usually misleading for tagging.
• In multi-label tagging, micro-averaged scores reflect overall volume, while macro-averaged scores give each label equal weight and expose rare labels that perform badly.
• A gold-standard sample is only as good as the agreement between the people who built it, so measure inter-annotator agreement with a statistic such as Cohen's kappa before trusting it.
• Confidence thresholds trade precision against recall, and should be set per label on held-out data, with uncertain cases routed to human review.
• Tagging quality drifts as content, language and the taxonomy change, so re-run the gold set after every model or taxonomy change and monitor review overrides continuously.

Precision, recall and F1, with a worked example

For a single label, every item falls into one of four groups:

• True positive (TP): the model applied the tag, and it is correct.
• False positive (FP): the model applied the tag, but it is wrong.
• False negative (FN): the tag should have been applied, but the model missed it.
• True negative (TN): the tag does not apply, and the model correctly left it off.

The scikit-learn documentation defines precision as TP / (TP + FP), the classifier's ability not to label a negative item as positive, and recall as TP / (TP + FN), its ability to find all the positive items. F1 is the harmonic mean of the two, which simplifies to 2TP / (2TP + FP + FN).

Suppose a newsroom checks the label "Elections" on a gold sample of 200 articles. Reviewers agree that 40 of them are about elections. The model applies the tag to 45 articles, of which 34 are correct.

• Precision = 34 / 45 = 75.6%: about one in four "Elections" tags is wrong.
• Recall = 34 / 40 = 85.0%: the model found most election stories but missed six.
• F1 = 68 / 85 = 0.80.
• Accuracy = (34 + 149) / 200 = 91.5%, which sounds better than the tag deserves.

Accuracy flatters tagging because most labels do not apply to most items. Google's Machine Learning Crash Course gives the extreme case: if the positive class appears 1% of the time, a model that always predicts negative scores 99% accuracy while being useless. Report precision and recall instead.

When one kind of error costs more, use the F-beta score: a beta below 1 favours precision, which suits public-facing tags; a beta above 1 favours recall, which suits tags used to find research material.

Multi-label evaluation: micro versus macro averaging

Real tagging is multi-label: one article can be about politics, housing and a particular town. scikit-learn treats a multi-label problem as a collection of binary problems, one per label, and then averages. The choice of average changes the story:

• Micro pools the TP, FP and FN counts across all labels before calculating, so each item-label pair counts equally. Frequent labels dominate.
• Macro calculates the score for each label and takes the plain mean, so every label counts equally. scikit-learn notes this highlights infrequent labels, but can over-emphasise their typically low performance if not all labels matter equally.
• Weighted averaging weights each label by its frequency; samples averaging scores each item, then averages across items.

A three-label example shows why you need both micro and macro:

The micro F1 of 0.89 looks healthy because politics and sport dominate the counts. The macro F1 of 0.73 reveals that the obituaries label misses seven in ten obituaries, a problem the micro figure hides.

Two cautions. First, macro F1 has two definitions: the mean of per-label F1 scores (0.73 here, and scikit-learn's method) or the harmonic mean of macro precision and macro recall (0.75 here). Opitz and Burst showed that the two can differ substantially and even rank models differently, so state which one you report. Second, item-level measures answer a different question: subset accuracy credits an item only if its whole tag set matches exactly, while Hamming loss and the Jaccard index give partial credit. Always keep the per-label table; it is where fixes come from.

Building a gold-standard sample

1. Write the label guidelines first. Each label needs a definition, a scope note and examples of what is in and out. These usually come straight from your taxonomy (see our guide to building a content taxonomy).
2. Sample deliberately. Combine a random sample, which gives honest overall figures, with a stratified top-up that guarantees enough examples of rare labels, older decades and each content type.
3. Size it for the precision you need. As a rough guide using the normal approximation, a precision of 85% measured on 100 tagged items has a 95% margin of error of about ±7 percentage points; on 400 items, about ±3.5. Rare labels need enough positive examples of their own before their recall means anything.
4. Annotate independently and blind. At least two people label each item without seeing the model's suggestions.
5. Adjudicate disagreements with a third reviewer, and update the guidelines when one reveals an unclear definition.
6. Keep it separate. Never use gold items as examples in prompts or training data, and version the gold set alongside the taxonomy release it was built for.

Our guide to data quality for AI datasets applies the same discipline to whole datasets.

Inter-annotator agreement and Cohen's kappa

If two careful people cannot agree on a label, the gold standard itself is uncertain. Raw percentage agreement overstates consistency because some agreement happens by chance; McHugh's review in Biochemia Medica notes that it does not account for raters guessing.

Cohen's kappa corrects for chance: kappa = (observed agreement − expected chance agreement) / (1 − expected chance agreement).

Two editors label 100 articles as "Opinion" or not. They both say yes on 20, both say no on 65, and disagree on 15 (editor A says yes on 10 that B rejects; B says yes on 5 that A rejects).

• Observed agreement = (20 + 65) / 100 = 0.85.
• Editor A says yes 30% of the time and B 25%, so chance agreement = 0.30 × 0.25 + 0.70 × 0.75 = 0.60.
• Kappa = (0.85 − 0.60) / (1 − 0.60) = 0.625.

So 85% raw agreement becomes a kappa of 0.625, and how to read that depends on the scale. Cohen's commonly cited bands call 0.61–0.80 substantial agreement; McHugh argues they may be too lenient for health research, and on her stricter scale any kappa below 0.60 indicates inadequate agreement. The scikit-learn documentation says scores above 0.8 are generally considered good agreement. McHugh's advice is to calculate both percentage agreement and kappa.

The scikit-learn documentation adds three points: kappa is intended to compare human annotators, not a classifier against ground truth; for multi-label data, compute it per label; and it covers two annotators only. For three or more, use a multi-rater statistic such as Fleiss' kappa, available in the statsmodels library. When agreement on a label is low, fix the definition and re-annotate before measuring the model on it.

Confidence thresholds and trade-offs

Most tagging models output a score, and a tag is applied when the score passes a threshold. scikit-learn's documentation separates the two steps: estimating a probability is a statistical problem, while deciding what to do with it is a decision problem. It says the default cut-off of 0.5 is not ideal for most use cases, that the threshold should be tuned for a metric that fits the use case, and that the same data should never be used both to train the model and to tune the threshold.

Google's Machine Learning Crash Course describes the trade-off: raising the threshold tends to reduce false positives and increase false negatives, so precision and recall often move in opposite directions. An illustrative curve for one label might look like this:

A common pattern is three bands per label: apply the tag automatically above a high threshold, send items between two thresholds to human review, and discard below the lower one. Set thresholds per label, because labels differ in cost: a public-facing "Obituaries" tag deserves a high precision bar, while an internal research tag can favour recall.

Check that scores mean what they say: group gold items by score band and count how many tags in each band are correct. If tags scored 0.9 or above are right only 80% of the time, adjust the thresholds.

Human-in-the-loop review and monitoring drift

• Route the right items. Send uncertain scores, rare labels, sensitive topics and new content types to reviewers; leave confident, common tags automatic.
• Show the evidence: the passage that triggered each tag.
• Make decisions data. Every accept, reject or replacement is a labelled example. A label that reviewers keep rejecting needs a better definition, a higher threshold or both.
• Spot-check the automatic path. Review a small weekly random sample of auto-applied tags to measure their precision directly.
• Check the reviewers. Periodically double-review a sample to confirm decisions are consistent.

In a newsroom this loop usually sits in the CMS as suggested tags that journalists accept or reject (see our guide to AI in the newsroom).

Quality also changes over time. Lu and colleagues' review of concept drift in IEEE Transactions on Knowledge and Data Engineering defines it as unforeseeable changes in the underlying distribution of streaming data over time. In archives, drift comes from new topics and names, changes in house style, newly digitised material with OCR errors, taxonomy releases and model or prompt changes. Watch for sudden shifts in how often each label is applied, rising override rates and falling scores on a fresh quarterly gold sample, and re-run the same gold set after every model, prompt or taxonomy change, like a regression test in software.

A practical evaluation plan

How Druid Learning helps

Druid Learning is an AI archival processing platform, based in Dublin and New York, that turns content archives into AI-ready datasets. Its Standardization Tool provides real-time content tagging and labelling. For NPR, Druid applied 20+ labels across 8 dimensions to more than 2 million assets, with suggested tags and related stories for journalists. For the Netherlands Cancer Institute, it enriched 20,000+ research samples with standardised metadata and delivered 100k+ structured data points, each with a confidence score.

Frequently asked questions

What is a good F1 score for AI tagging?

There is no universal threshold. A useful benchmark is how well your own trained annotators agree on the same labels, because a gold standard that people dispute limits how precisely any model can be scored. Judge scores per label and by purpose: a public-facing label needs high precision, while a label used to find archive material for research can accept lower precision in exchange for higher recall. Compare against the previous system, not an abstract target.

What is the difference between micro and macro averaging?

Micro averaging pools true positives, false positives and false negatives across all labels before calculating precision, recall or F1, so frequent labels dominate the result. Macro averaging calculates each label's score separately and takes the plain mean, so every label counts equally. A high micro score with a low macro score usually means a few common labels perform well while some rarer labels perform badly. Report both, plus the per-label table.

How large should a gold-standard sample be?

Large enough that the margin of error is smaller than the differences you care about. Using the normal approximation, a precision of 85% measured on 100 tagged items has a 95% margin of error of about seven percentage points; on 400 items it is about three and a half. Rare labels need their own minimum number of positive examples, so add a stratified top-up to a random sample.

What is Cohen's kappa and why does it matter for tagging?

Cohen's kappa measures agreement between two annotators after correcting for the agreement expected by chance. It ranges from −1 to 1, where zero means no better than chance. It matters because a gold standard built by people who often disagree cannot reliably score a model. scikit-learn notes that scores above 0.8 are generally considered good agreement, while McHugh argues that values below 0.60 indicate inadequate agreement.

How often should AI tagging be re-evaluated?

Re-run the full gold set whenever the model, prompt or taxonomy changes, in the same way software teams run regression tests before a release. Between changes, spot-check a small random sample of automatically applied tags each week, track review override rates per label, and tag a fresh gold sample every quarter, because new topics, names and content sources gradually shift what the model sees.

Sources

• Metrics and scoring: quantifying the quality of predictions — scikit-learn
• Tuning the decision threshold for class prediction — scikit-learn
• Classification: Accuracy, recall, precision, and related metrics — Google for Developers
• Interrater reliability: the kappa statistic (McHugh, 2012) — Biochemia Medica
• Macro F1 and Macro F1 (Opitz and Burst, 2019) — arXiv
• Learning under Concept Drift: A Review (Lu et al.) — arXiv
• statsmodels.stats.inter_rater.fleiss_kappa — statsmodels