Multilingual Archives: Language Detection, Translation and AI

Preparing a multilingual archive for AI means knowing reliably which language each document and passage is in, storing all text in a consistent Unicode form, and then deciding whether to translate content or use models that handle many languages directly. With those foundations in place, a single search can find relevant material whatever language it was written in.

Most archives are more multilingual than their catalogues suggest. A broadcaster may hold programmes in two official languages, a publisher may produce bilingual textbooks, and a newspaper will have quoted people in their own words for decades. If the catalogue records one language per item, or none, much of that material is hard to find.

This guide covers language identification, mixed-language documents, scripts and Unicode normalisation, OCR beyond the Latin alphabet, machine translation versus multilingual models, cross-lingual search, metadata in more than one language, and quality checks for low-resource languages.

Key takeaways

• Language identification is reliable for long passages in widely used languages but much weaker for short text, closely related languages and low-resource languages, so its output needs checking.
• Normalise text to a single Unicode form, usually NFC, before indexing, deduplicating or comparing strings.
• OCR quality depends on using models for the right language and script, including historical typefaces and vertical or right-to-left text.
• Multilingual embedding models allow cross-lingual search without translating everything, while machine translation remains useful for readers and for languages that models cover poorly.
• Record language at both document and passage level using standard BCP 47 tags, and store labels in each language as separate, linked values.

Language identification

Language identification (often shortened to language ID or LID) is the task of deciding which language a piece of text is written in. Most tools are classifiers trained on character patterns: sequences such as 'ij' in Dutch or 'ñ' in Spanish are strong signals, and the model learns thousands of them.

Widely used open tools include fastText's language identification models, which recognise 176 languages. They come in two sizes, a 126MB model and a compressed version of under 1MB, were trained on Wikipedia, Tatoeba and SETimes, and expect UTF-8 input. Coverage has since grown considerably: GlotLID (Kargaran et al., Findings of EMNLP 2023) identified 1,665 languages in the version evaluated, outperforming baselines including Google's CLD3 and fastText's 176-language model when balancing F1 against false positives; its 2024 v3.0 release includes over 2,000 labels.

Benchmark scores can overstate real-world accuracy. Caswell et al. (COLING 2020) trained models covering up to 1,629 languages that scored well on held-out test sets, yet when people reviewed web-crawled corpora built with them, accuracy for many low-resource languages was only around 5%. The causes they identified were domain mismatch, class imbalance (a huge language misclassified even rarely swamps a small one), similarity between related languages and models that were not expressive enough. For a sample of languages, word-list filters raised median precision from 5.5% to around 50%, and a semi-supervised LangID model to 71.2%.

Practical rules for archives:

• Detect at paragraph level as well as document level. A document-level label hides quotations and parallel texts.
• Do not trust short strings. Titles, captions and tags give a classifier little to work with; use the surrounding document or catalogue record instead.
• Keep the confidence score. Route low-confidence results to review rather than accepting the top guess.
• Restrict to plausible languages. If a collection is known to contain three languages, re-check predictions outside that set.
• Watch closely related languages. Caswell et al. found typical models often confuse Nigerian Pidgin with English; such pairs need specialist checks.

Mixed-language documents

Many archive items contain more than one language, in different patterns:

The aim is that every downstream system can tell which text is in which language. In HTML and XML, the lang and xml:lang attributes mark language on any element, and child elements inherit it unless they declare their own, so a quoted passage can be tagged without relabelling the page. In a dataset, store a primary language for each document, a list of all languages present, and a language field on each chunk.

Scripts, encodings and Unicode normalisation

Before any AI processing, all text should be in one encoding, UTF-8, and one normalisation form. Older collections often contain legacy encodings, and text converted with the wrong one produces mojibake, such as 'café' for 'café'. Scan for these patterns early.

Even correctly encoded text can represent the same visible characters in different ways. 'é' can be stored as one precomposed character or as 'e' followed by a combining accent. The two look identical but are different strings, so search and duplicate detection may treat them as different. Unicode Standard Annex #15 defines four normalisation forms that resolve this:

The compatibility forms (NFKC and NFKD) go further than the canonical ones. They turn the 'fi' ligature into 'f' and 'i', half-width katakana into standard katakana, a superscript '⁵' into '5', and the historical long s, 'ſ', into 's'. That is helpful for matching but loses information, and the annex warns that these forms must not be blindly applied to arbitrary text. A sensible pattern is to store the original text in NFC and build a separate NFKC, case-folded version for search.

Two script issues deserve checking on a sample of documents. Right-to-left scripts such as Arabic and Hebrew can come out of some extraction tools in the wrong order, so confirm that extracted text reads correctly. Languages written without spaces between words, such as Chinese, Japanese and Thai, need tokenisers that understand them, because tools that split on spaces will treat whole sentences as single words.

OCR beyond the Latin alphabet

Optical character recognition models are trained for particular languages and scripts, so running a model for the wrong language or script produces poor text. The open-source Tesseract OCR engine illustrates the options:

• trained models for more than 100 languages;
• script-level models, such as Arabic, Devanagari and Cyrillic, each covering several languages that share a script;
• vertical-text models for Chinese, Japanese and Korean scripts;
• models for Fraktur, the blackletter type used in older German printing;
• models for historical languages including Middle English, Middle French, Ancient Greek and Latin;
• an orientation and script detection module for identifying a page's script and rotation before recognition.

In practice, identify each page's language and script before OCR (from catalogue data or a detection pass), run the matching model or combination of models, and measure accuracy separately for each language. Watch diacritics: an OCR model that drops accents can turn one word into another while the text still looks plausible. The guide to OCR for archives covers accuracy measurement and quality control in detail.

Machine translation or multilingual models?

There are two broad ways to make content in many languages usable by AI: translate it into a common language, or use models that work across languages directly.

Machine translation has improved substantially for languages that were long neglected. The No Language Left Behind project (NLLB Team, 2022) released open-source models covering 200 languages, evaluated on FLORES-200, a human-translated benchmark spanning more than 40,000 translation directions, and reported a 44% BLEU improvement relative to the previous state of the art. The team also built a toxicity benchmark covering all FLORES-200 languages to check translation safety.

Multilingual embedding models take the other route, mapping text in different languages into a shared space where translations sit close together. LaBSE (Feng et al., ACL 2022), for example, covers 109+ languages and was designed for cross-lingual retrieval and for finding pairs of sentences that are translations of each other. On the Tatoeba benchmark across 112 languages it reached 83.7% retrieval accuracy, against 65.5% for the earlier approach it was compared with.

Whatever you choose, keep the original text as the authoritative version, and mark machine translations clearly in metadata, with the system and date used.

Cross-lingual search

With multilingual embeddings, a query in English can retrieve a relevant passage in German or Irish without translation. Keyword search does not work this way: BM25 matches words, so it needs language-specific processing such as tokenisation and stemming, and only finds matches in the query's language. Combining multilingual vector search with per-language keyword indexes covers both needs; see embeddings and vector search. Test with the same questions in each language, and check that results do not systematically favour the collection's largest language.

Metadata in multiple languages

Language metadata should use standard codes. BCP 47, the IETF standard behind the lang attribute, builds tags from subtags in a fixed order: a language code, then optionally a script and a region. Examples from the W3C's guidance include en-GB for British English, zh-Hans for Chinese in simplified script and es-419 for Latin American Spanish. The W3C's golden rule is to keep tags as short as possible: add a script or region only when it makes a distinction you need.

For descriptive labels such as subjects, topics and curriculum terms, store each concept once with a stable ID and attach a label in each language, rather than keeping separate, unlinked vocabularies. A generic example:

{"concept_id": "topic-0412", "labels": {"en": "Local elections", "fr": "Élections locales"}, "source": "editorial taxonomy v3"}

Where an official or authoritative translation of a term exists, such as a subject name in a national curriculum, use it rather than a machine translation, and record its source. Labels that are machine-translated should be reviewed by a speaker before they reach users. The guide to building a content taxonomy covers how to structure the vocabulary itself.

Low-resource languages and quality checks

Languages with little digital text are where automated tools are weakest and where archives often hold the most distinctive material. The GlotLID authors list the recurring problems: incorrect language labels in source corpora, text in high-resource languages leaking into low-resource collections, confusion between closely related languages, and uncertainty about whether to label a macrolanguage or its individual varieties.

A practical checklist for each language in the collection:

1. Count it. Report documents and tokens per language so small languages are visible rather than averaged away.
2. Sample and read. Have a speaker review a random sample labelled with each language, checking language, script and readability.
3. Check characters and words. Text labelled as a language should mostly use its expected characters, and checking for its common words, as Caswell et al. did, quickly flags mislabelled text.
4. Measure tools per language. Report OCR, transcription and retrieval accuracy for each language, not as a single figure.
5. Record varieties. Where it matters, record dialect or historical spelling, and keep original spellings alongside modernised versions.

These checks fit within a wider data quality process for AI datasets.

How Druid Learning helps

Druid Learning is an AI archival processing platform, based in Dublin and New York, that turns content archives into AI-ready datasets and handles content in 140+ languages. It processes formats including OCR scans, PDF, XML, EPUB, WAV, MP3 and MP4; Archive Optimization turns an archive in any format into an indexed, labelled, AI-ready dataset, and the Standardization Tool provides real-time content tagging and labelling. For CJ Fallon, an Irish educational publisher, 50,000 resources were processed and mapped to the curriculum.

Frequently asked questions

How accurate is automatic language detection?

For paragraphs of text in widely used languages, modern language identification tools are usually very accurate. Accuracy drops sharply for short strings such as titles, for closely related languages and for low-resource languages. One study found that corpora built with models that scored well on benchmarks had only around 5% accuracy for many low-resource languages when reviewed by people, so automated labels should be sampled and checked.

Should I translate my archive before indexing it for AI?

Not necessarily. Multilingual embedding models can search across languages without translation, preserving the original text and avoiding the cost of translating everything. Translation remains useful when staff or users need to read content in another language, or when models handle a particular language poorly. If you do translate, keep the original as the authoritative version and label machine translations clearly in metadata.

What is Unicode normalisation and why does it matter?

Unicode normalisation converts text to a standard representation, because the same visible character can be stored in more than one way, such as an accented letter written as one character or as a letter plus a combining accent. Without normalisation, identical-looking words may not match in search or deduplication. NFC is the usual form for stored text, while NFKC is sometimes used for search keys.

How do I record the language of mixed-language documents?

Record language at more than one level. Store a primary language for the document, a list of every language it contains, and a language value for each paragraph or chunk. Use standard BCP 47 tags such as en or zh-Hans. In HTML and XML, the lang and xml:lang attributes can mark quoted passages in another language without changing the label of the whole document.

Can AI tools handle low-resource languages?

Increasingly, but unevenly. Translation models now cover 200 languages and language identification tools more than 1,600, yet quality is usually lower for languages with little digital text, and benchmark results can overstate real-world accuracy. For any low-resource language in an archive, measure OCR, detection and retrieval accuracy separately, and have speakers review samples before relying on the output.

Sources

• Language Identification — fastText
• GlotLID: Language Identification for Low-Resource Languages — Kargaran et al., Findings of EMNLP 2023 (arXiv)
• Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus — Caswell et al., COLING 2020 (arXiv)
• Unicode Standard Annex #15: Unicode Normalization Forms — Unicode Consortium
• Languages/Scripts Supported in Different Versions of Tesseract — Tesseract Documentation
• No Language Left Behind: Scaling Human-Centered Machine Translation — NLLB Team (arXiv)
• Language-agnostic BERT Sentence Embedding — Feng et al. (arXiv)
• Language Tags in HTML and XML — W3C Internationalization