Finding Duplicates and Near-Duplicates in Large Content Archives

Content archives contain duplicates because the same text is syndicated, reprinted, re-issued in new editions, saved in several formats and scanned more than once. Exact copies can be found cheaply by hashing normalised text. Near-duplicates, which differ by a headline, a byline, a corrected paragraph or a scattering of OCR errors, need similarity methods: shingling combined with MinHash and locality-sensitive hashing, or SimHash fingerprints.

Deduplication matters more once an archive becomes an AI-ready dataset. Repeated text clutters search results and inflates counts, and in AI training it encourages models to memorise passages and contaminates the sets used to evaluate them.

This guide covers where duplicates come from, how the main detection methods work, what research says about deduplication and model quality, and a practical workflow for setting thresholds.

Key takeaways

• Duplicates are a predictable result of syndication, reprints, editions, format conversions and repeated digitisation, so expect them in any large archive.
• Hash normalised text to find exact duplicates, and use MinHash with locality-sensitive hashing, or SimHash, to find near-duplicates at scale.
• Treat fast detection as candidate generation and confirm candidate pairs with a direct similarity measure before acting on them.
• Research on language model training shows deduplication reduces memorisation and train-test overlap, but removing more data does not automatically produce a better dataset.
• Flag and group duplicates with a recorded canonical copy instead of deleting them, so each user can apply their own policy.

Why archives contain so many duplicates

Duplication in an archive is rarely a mistake; it records how content was produced and reused:

• Syndication and wire copy. An agency story may run in dozens of titles, each with its own headline, a trimmed ending or an added local paragraph.
• Re-issues and editions. Books are reprinted and revised; a new textbook edition may rewrite a fifth of the text and leave the rest untouched.
• Versions. Online articles are updated and corrected, and content management systems keep print and web versions of the same story.
• Multiple formats. One book may exist as InDesign files, a print PDF, an EPUB and an XML feed, each yielding slightly different text.
• Repeated digitisation. Pages scanned once from microfilm and again from bound volumes, or imported twice during a system migration, produce near-identical text with different OCR errors.
• Templates and boilerplate. Sports results, weather, share prices, standard notices and recurring columns produce many documents that are mostly the same. Lee and colleagues observed that most duplicates they saw in training corpora were automatically generated pages, such as the outcomes of sports games.

Each source implies a different relationship. Two scans of one page are the same item with different errors; a syndicated story and its local rewrite are related but not identical. Good deduplication records which relationship applies, not just a yes or a no.

Exact duplicates: normalise, then hash

Exact duplicates are the easy case. As Gurmeet Singh Manku and colleagues put it in a 2007 paper on web crawling, exact duplicates are easy to identify with standard checksumming techniques. In an archive it pays to hash at two levels:

1. File hash. A cryptographic hash such as SHA-256 of each file's bytes finds identical files created by repeated imports or copied folders, but misses identical content saved in different files.
2. Normalised text hash. Extract the text, normalise it (consistent Unicode form, lower case, collapsed whitespace, markup and page furniture removed), then hash the result. This finds the same text held as a PDF and as XML.

Normalisation decides what counts as "the same", so choose it deliberately and record the rules. Aggressive normalisation, such as stripping numbers, can merge items that genuinely differ: two short match reports whose only difference is the score.

A third exact method works below the document level. Lee and colleagues built a suffix array over an entire corpus to find repeated passages of at least 50 tokens, and found a single 61-word English sentence repeated more than 60,000 times in the C4 web dataset. In an archive, the same idea finds boilerplate paragraphs, standard disclaimers and recycled standfirsts worth stripping or marking before any near-duplicate pass.

Near-duplicates: shingling, MinHash and LSH

Shingles and resemblance

The foundational work comes from Andrei Broder, who worked on the AltaVista search engine at Digital's research labs in the mid-1990s. Soon after its first internal prototype was deployed, a colleague complained that half the first page of results was often the same document in many variants. Broder's answer, set out in his 1997 paper On the Resemblance and Containment of Documents and later papers, was to turn each document into a set and compare sets.

The set is built by shingling: taking every contiguous sequence of w words. Broder's example is the 4-shingling of "a rose is a rose is a rose", which gives three distinct shingles: (a, rose, is, a), (rose, is, a, rose) and (is, a, rose, is). The resemblance of two documents is the Jaccard similarity of their shingle sets: the number of shingles they share divided by the number in either. It runs from 0 (nothing shared) to 1 (identical sets).

MinHash

Comparing full shingle sets is expensive, so MinHash compresses each set into a short signature. Apply a random hash function to every shingle and keep the smallest value. As Broder showed, if the hash behaves like a random permutation, the probability that two documents share the same minimum equals their Jaccard similarity. Repeat with k independent hash functions (Broder's example uses 100), and the fraction of positions where the two signatures match estimates the similarity. Each position is a yes-or-no match, so the estimate behaves like a sampled proportion: with 100 hash functions its standard error is at most 0.05.

Locality-sensitive hashing

Even with signatures, comparing every pair is impractical: a million documents form roughly 500 billion pairs. Locality-sensitive hashing (LSH) avoids this. Split each signature into b bands of r values. Two documents become a candidate pair if all r values match in at least one band. For two documents with true similarity s, the probability of that is 1 − (1 − s^r)^b, an S-shaped curve whose steep section you place at your chosen threshold.

For the FineWeb web dataset, Guilherme Penedo and colleagues used word 5-grams and 112 hash functions split into 14 bands of 8, targeting documents at least 75% similar. With those settings, the probability that a pair is flagged is about 56% at 0.70 similarity, 77% at 0.75, 92% at 0.80 and 98.8% at 0.85. The threshold is soft: the datasketch library's documentation stresses that LSH returns candidates, with false positives and false negatives, and recommends re-checking results if you need an exact cut-off.

SimHash

SimHash, built on a technique Moses Charikar published in 2002, takes a different route. Each weighted feature of a document (for example a word or phrase) is hashed to a string of bits. For every bit position, the algorithm adds the feature's weight where its hash has a 1 and subtracts it where it has a 0, then sets the final bit according to the sign of the total. Similar documents end up with fingerprints that differ in only a few bits. Building near-duplicate detection for Google's web crawler, Manku, Jain and Das Sarma found that, for a repository of 8 billion web pages, 64-bit SimHash fingerprints with a threshold of 3 differing bits were reasonable. One small fingerprint per document makes SimHash well suited to checking new items against a very large index as they arrive.

Choosing a method

Embedding-based similarity, explained in our guide to embeddings and vector search, is useful for finding stories about the same event, but that is topic clustering, not deduplication.

Why deduplication matters for AI training data

Key evidence comes from Deduplicating Training Data Makes Language Models Better by Katherine Lee and colleagues, published at ACL 2022. Using MinHash followed by an edit-similarity check, they found near-duplicates made up between 3.04% of training examples in C4 and 13.63% in RealNews, a news corpus, against only 0.39% in Wiki-40B. Most near-duplicate clusters in C4 were simple pairs, but one cluster held 250,933 examples. Models trained on deduplicated data emitted memorised text about ten times less often, needed fewer training steps to reach the same or better accuracy, and could be evaluated more honestly, because train-test overlap affected more than 4% of the validation sets of standard datasets.

The FineWeb team found a useful counterpoint. Deduplicating globally across 96 Common Crawl snapshots removed as much as 90% of the data from the oldest snapshots, yet models trained on the result barely improved on undeduplicated data. In one older snapshot, the text that survived was of worse quality than the text removed. Deduplicating each snapshot separately gave better results. The authors suggest the main benefit of deduplication comes from removing large clusters of duplicates.

For archive owners, three lessons follow:

• In Lee's figures the news corpus had far more near-duplication than the curated Wikipedia corpus, so measure the rate in your own archive rather than guess.
• Duplicates that cross training and evaluation sets distort results, so duplicate groups should never be split between them.
• More aggressive deduplication is not automatically better. A recipient may want a different policy, which is another reason to flag rather than delete. Duplicate rates are also a core measure in a data quality report.

A practical workflow

1. Define the unit and the question. Decide whether you are deduplicating articles, pages or chapters, and what "the same" means: same item, same text, or same story.
2. Extract and normalise text. Record the normalisation rules and their version.
3. Run the exact pass. Hash files, then normalised text; group the matches.
4. Mark boilerplate. Find lines and paragraphs repeated across many documents and strip or tag them, so unrelated items do not look similar because of a shared masthead or footer.
5. Generate near-duplicate candidates. Shingle the text (word 5-grams were used in both studies above), compute MinHash signatures and tune LSH to your threshold.
6. Verify candidates. Compute exact Jaccard or edit similarity for each candidate pair. Lee and colleagues required actual Jaccard similarity above 0.8, then edit similarity above 0.8.
7. Cluster. Join verified pairs into groups. Watch for chaining, where A resembles B and B resembles C but A and C are quite different, and review unusually large groups.
8. Choose a canonical copy by rule, such as the earliest publication, the best OCR, the most complete metadata or the preferred edition, and record the rule.
9. Store the result as data. Add a cluster ID, a canonical flag, the relationship type and the similarity score to each record.
10. Review the boundary and document it. Check sample pairs just above and below the threshold, adjust, and report the parameters and duplicate rates in the dataset documentation.

Choosing thresholds

There is no universal threshold. Label a few hundred candidate pairs from across the similarity range, decide which ones your organisation would call duplicates, and set the threshold where that judgement changes. Three content-specific cautions apply:

• Short items behave badly. Captions, news briefs and results tables have few shingles, so similarity estimates are unstable. Lee and colleagues found 90% of documents in one corpus were under 50 tokens, too short for their substring method to consider. Handle short items separately, often with exact matching only.
• Containment differs from resemblance. When a short wire story sits entirely inside a longer local version, Jaccard similarity can be modest even though one contains the other. Broder's 1997 paper defines a separate containment measure: the shared shingles divided by the shingles of the document that may be contained. Use it when excerpts matter.
• Different content types need different settings. Editions of a textbook, syndicated news and repeated scans of one page each need their own threshold, tested on their own samples.

How Druid Learning helps

Druid Learning is an AI archival processing platform, based in Dublin and New York, that turns content archives into AI-ready datasets. Archive Optimization turns archives in formats including OCR scans, PDF, XML, IDML, INDD, EPUB, WAV, MP3 and MP4 into an indexed, labelled dataset. The platform has processed more than 10 million assets, including more than 2 million for NPR and 50,000 educational resources for CJ Fallon, mapped to the curriculum, and handles content in more than 140 languages.

Frequently asked questions

What is the difference between a duplicate and a near-duplicate?

An exact duplicate is identical, byte for byte or once text has been normalised, and can be found by hashing. A near-duplicate is substantially the same content with small differences, such as a new headline, a trimmed paragraph, a correction or different OCR errors. Near-duplicates need similarity methods such as MinHash or SimHash, combined with a threshold you choose.

How does MinHash detect near-duplicate documents?

MinHash breaks each document into overlapping word sequences called shingles, hashes them, and keeps the smallest hash value. The chance that two documents share that minimum equals their Jaccard similarity: the proportion of all their distinct shingles that appear in both. Repeating this with many hash functions gives a compact signature, and locality-sensitive hashing then finds documents whose signatures match closely without comparing every possible pair.

What similarity threshold should we use?

There is no universal answer. Published research offers reference points, such as a 0.8 edit-similarity cut-off and a target of 75% shingle similarity, but the right threshold depends on what your organisation means by 'the same'. Label a few hundred candidate pairs across the similarity range, see where judgements change, and set separate thresholds for different content types.

Should we remove duplicates before sharing an archive with an AI company?

Usually it is better to flag them than remove them. Group duplicates, mark a canonical copy by a documented rule, and include the cluster IDs and similarity scores in the dataset. The recipient can then apply their own policy, which matters because research shows that the most aggressive deduplication does not always produce the best training data.

Why does deduplication improve AI models?

Repeated text gets extra weight in training, encouraging models to memorise and reproduce it. Research by Lee and colleagues found that models trained on deduplicated data emitted memorised text about ten times less often and reached the same or better accuracy in fewer training steps. Deduplication also stops test material leaking into training data, which keeps evaluation honest.

Sources

• On the Resemblance and Containment of Documents — Andrei Z. Broder, SEQUENCES 1997
• Identifying and Filtering Near-Duplicate Documents — Andrei Z. Broder, CPM 2000
• Similarity Estimation Techniques from Rounding Algorithms — Moses S. Charikar, STOC 2002
• Detecting Near-Duplicates for Web Crawling — Manku, Jain and Das Sarma, Google
• Deduplicating Training Data Makes Language Models Better — Lee et al., arXiv
• The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale — Penedo et al., arXiv
• MinHash LSH — datasketch documentation