Making Audio and Video Archives Searchable

To make an audio or video archive searchable, turn what is said into time-coded text with automatic speech recognition (ASR). Mark who is speaking with speaker diarisation. For video, split the footage into shots and read any on-screen text. Then describe each recording, and each segment within it, with standard metadata. A search hit then takes the user to the exact moment in the recording, not just to a file that might contain the answer.

This guide covers each step: the digital files you need first, how modern speech-to-text works and how word error rate measures it, diarisation and timestamps, caption formats (SRT and WebVTT), and accents, multiple languages and poor audio. On the video side it covers shot detection, OCR of on-screen text and the EBUCore and PBCore metadata standards. It ends with a practical workflow and quality-control plan.

Key takeaways

• Transcription quality depends on the audio you feed it, so start from good preservation files and make separate access copies for processing.
• Word error rate is the standard accuracy measure, but you must normalise text before comparing it, or formatting differences will be counted as errors.
• Large speech models are robust but can hallucinate, repeat or drop words in long recordings, and accuracy varies widely between languages.
• Store word-level timestamps and speaker turns, not just a block of text, so search results, captions and quotes can link back to the moment they come from.
• Describe assets and segments with an established AV metadata standard such as EBUCore or PBCore so the results are exchangeable and durable.

Start with good digital files

Many audio and video archives start on physical carriers: tape, disc and film. A 2021 Digital Preservation Coalition guidance note on preserving audio, citing IASA guidelines among others, gives 96 kHz as the recommended sampling rate for preservation audio, with 48 kHz the recommended minimum, and says bit depth should be at least 24 bits to capture the full dynamic range. For recordings that are already digital, such as CDs or DAT tapes, it advises keeping the native sampling rate and bit depth.

For the container, the same note describes Broadcast Wave Format (BWF, standardised as EBU Tech 3285) as an extension of WAVE that can embed metadata, making the file self-describing. Numerous organisations consider it a preferred preservation format, with a limit of 4 GB of audio data per file.

Keep that preservation master untouched and create access copies for processing. Speech models don't need preservation-grade audio. Whisper, for example, resamples all input to 16,000 Hz. Do record which master each access copy came from, so every transcript can be traced back.

Speech-to-text and how it is measured

Modern ASR systems are neural networks trained on large amounts of paired audio and text. A well-known example is OpenAI's Whisper, described in the 2022 paper Robust Speech Recognition via Large-Scale Weak Supervision. The authors trained on 680,000 hours of audio with transcripts collected from the internet. Of that, 117,000 hours covered 96 languages other than English, and 125,000 hours were translation data into English. They report that the models are often competitive with fully supervised systems without fine-tuning on each benchmark, and approach human accuracy and robustness.

Word error rate

Accuracy is measured with word error rate (WER). WER counts the minimum number of word substitutions, deletions and insertions needed to turn the system's transcript into a correct reference, divided by the number of words in the reference.

Taken literally, that is three errors in nine words ("on" to "in", "fourteenth" to "14th", "May" to "may"), a WER of 33%. Only one of them is a real mistake. The Whisper paper makes this point: WER penalises innocuous differences in transcript style. Its authors normalised text before scoring and saw WER fall by up to 50% on some datasets, usually because of quirks in how the reference transcripts were written. Agree a normalisation policy before measuring: case, punctuation, numbers, spelling variants and filler words.

Known failure modes

The same paper lists stubborn errors in long recordings. Models can get stuck in repeat loops, skip the first or last few words of a segment, or produce a "complete hallucination" unrelated to the audio. Whisper processes 30-second windows. Long recordings are transcribed window by window, so a mistake in one window can affect the next. The authors developed decoding heuristics to reduce these problems. Separately, the WhisperX paper reports that buffered long-form transcription is prone to drifting, hallucination and repetition. Its approach cuts the audio at speech boundaries found by voice activity detection before transcribing.

Practical consequence: check for text in silent or music-only passages, and for phrases repeated many times. Both are signs of hallucination.

Speakers, timestamps and captions

Speaker diarisation

Diarisation answers "who spoke when". It splits the audio into speaker turns and gives each turn a label (Speaker 1, Speaker 2), which you can later map to real names where known. It is usually a separate component from recognition. The Whisper paper notes that full systems often handle voice activity detection, diarisation and inverse text normalisation as separate components. Its authors also observed early models guessing speakers' names from context, almost always wrongly, and fine-tuned the models to stop it. Never treat a speaker name produced by a model as a fact without checking it.

Timestamps

Store timing at the finest level you can get. The WhisperX authors note that Whisper's per-utterance timestamps can be inaccurate and that word-level timestamps are not available out of the box. WhisperX adds them with forced phoneme alignment. Word-level timing lets a search interface jump to the exact word, lets editors pull accurate quotes, and lets you rebuild captions with different line lengths later.

Caption formats: SRT and WebVTT

The same caption in each format:

1
00:01:02,500 --> 00:01:05,000
The council will meet on the fourteenth of May.

WEBVTT

intro-01
00:01:02.500 --> 00:01:05.000
<v Chair>The council will meet on the fourteenth of May.

SRT was the basis for WebVTT. For archives, WebVTT is the better default: its encoding is defined, it can carry speaker names, and it can also hold chapter markers. Whichever you deliver, keep the richer underlying data (words, times, speakers, confidence) in your own store and generate caption files from it.

Accents, multiple languages and poor audio

Accuracy is not uniform. The Whisper paper reports that performance on a language is very well predicted by how much training data exists for it. Because its training data was English-heavy, most languages had less than 1,000 hours. For an archive, that means:

• Test per language and per collection. Measure WER on a sample of each language, accent, period and recording type you hold. A model that works well on studio news may struggle with regional dialect in oral history recordings.
• Detect language per segment, not per file. Interviews, broadcasts and meetings often switch language mid-recording. Our guide to multilingual archives covers detection and translation.
• Keep the original language. Whisper can also translate speech into English, but store a transcript in the spoken language alongside any translation, and label which is which.
• Pre-process difficult audio. Separate long silences and music beds with voice activity detection. The Whisper authors found it improved long-form results in their tests. Process each channel separately where speakers were recorded on separate tracks.
• Add vocabulary help. Names of people, places and organisations are where errors hurt most. Lists of expected names, and post-editing against an authority file, target the errors that matter most.
• Route the worst audio to people. Hiss, distortion, overlapping speech and distant microphones remain hard for automatic systems. Flag segments with low confidence or high estimated error for human transcription.

Video: shots, scenes and on-screen text

Video adds a visual timeline that is just as searchable as speech.

Shot detection finds the cuts and transitions in footage. The open-source PySceneDetect tool shows the main techniques:

• comparing colour differences between frames to catch fast cuts
• comparing those differences against a rolling average, rather than a fixed threshold, to adapt to the footage
• tracking average brightness to find fades in and out
• comparing histograms or perceptual hashes of frames

It can export timecodes, save the first and last frame of each scene, write edit decision lists and split the video into clips. Scenes, meaning groups of shots that belong together, such as one interview or one news item, are usually built on top of shots using speech, on-screen text and visual similarity.

Keyframes from each shot are the input for visual analysis:

• OCR of on-screen text, such as name captions (lower thirds), burnt-in subtitles, slides, signs and credits. These often give names, titles, places and dates that never appear in the speech. The same OCR quality issues apply as with documents. See our guide to OCR for archives.
• Visual labels for objects, settings and people, with confidence values, kept separate from human-verified descriptions.

Store every visual finding with its start and end time, so it can be combined with the transcript in search.

Metadata for audiovisual archives

Two established AV metadata standards are worth adopting rather than inventing your own.

• EBUCore (EBU Tech 3293) is the European Broadcasting Union's core metadata set. Version 1.10, published in April 2020, is available as a schema and as an RDF ontology. It lets items such as props, costumes, timed text, actions and emotions be linked to scenes, persons or characters.
• PBCore is a cataloguing standard for describing audiovisual content, originally developed by public broadcasting communities in the United States and since adopted by moving image archives. It also serves as a data model for collection management systems and as an import and export format between applications.

Whichever standard you choose, record metadata at two levels:

Segment-level records are what make an archive useful for search and for AI. A retrieval system can return the 40 seconds that answer a question, not a two-hour file. Our guide to chunking documents for retrieval explains why sensible segment boundaries matter.

A practical workflow and quality control

1. Inventory and digitise to preservation standards, and create access copies for processing.
2. Run voice activity detection and language identification to find speech and its language.
3. Transcribe with an ASR model suited to each language, keeping word-level timestamps and confidence.
4. Diarise to add speaker turns, then map labels to real names only where verified.
5. Process video: detect shots, extract keyframes, OCR on-screen text and add visual labels with times.
6. Describe assets and segments in EBUCore or PBCore, using controlled vocabularies for topics, people and places.
7. Generate outputs: WebVTT or SRT captions, searchable transcripts and segment-level records for search or AI datasets.

For quality control, have people transcribe a random sample from each collection and language. Measure normalised WER against it, and set acceptance thresholds by use. Search can tolerate more error than published captions or quotations. Also check:

• speaker turns and names on a sample
• timestamps at random points, with no more than a fraction of a second of drift
• caption files validated against the format, with readable line lengths
• silent and music-only passages for hallucinated text
• names, places and numbers, which deserve extra attention because they are error-prone and are often what users search for

How Druid Learning helps

Druid Learning is an AI archival processing platform, based in Dublin and New York, that turns content archives into AI-ready datasets. It processes audio and video formats including WAV, MP3 and MP4, as well as documents such as PDF, XML and OCR scans, and handles content in over 140 languages. Its Archive Optimization product turns archives in any format into an indexed, labelled dataset. For NPR, Druid Learning processed more than 2 million assets and applied over 20 labels across eight dimensions, supporting archive search by label.

Frequently asked questions

What is a good word error rate for archive transcription?

There is no single threshold, because it depends on the audio and the use. Measure WER on a hand-transcribed sample from your own collection after normalising case, punctuation and numbers. Without normalisation, formatting differences inflate the score: the Whisper authors saw WER fall by up to 50% on some datasets from normalisation alone. Searchable archives can tolerate more errors than published captions or quotations, which usually need human review.

What is the difference between SRT and WebVTT?

Both are plain-text caption formats that pair timecodes with text. SRT has no formal specification or defined character encoding, and uses a comma before milliseconds. WebVTT is a W3C specification. Files start with the word WEBVTT, must be UTF-8, and use a full stop before milliseconds. WebVTT can also carry speaker voice tags, chapter markers and time-aligned metadata, which makes it the better default for archives.

Does speech-to-text identify who is speaking?

Not on its own, in most systems. Identifying who spoke when is called speaker diarisation, and it is usually a separate component that splits audio into turns labelled Speaker 1, Speaker 2 and so on. Mapping those labels to real names needs a known voice sample or human review. The Whisper authors found early models guessed speaker names from context and were almost always wrong.

Why does automatic transcription sometimes invent text?

Speech models such as Whisper generate text one token at a time, guided by the text produced so far as well as the audio. On long recordings, silence, music or noise, they can get stuck in loops, skip words or produce text unrelated to the audio, which the Whisper authors call complete hallucination. Detecting speech segments before transcription reduces the problem. Quality control should check silent passages and repeated phrases.

Which metadata standard should I use for an audiovisual archive?

EBUCore and PBCore are the two established choices. EBUCore is the European Broadcasting Union's core metadata set, available as a schema and an RDF ontology. PBCore, which originated in US public broadcasting, is a cataloguing standard and data model since adopted by moving image archives and other media organisations. Choose the one your partners and systems already use, and record metadata both per recording and per time-coded segment.

Sources

• Robust Speech Recognition via Large-Scale Weak Supervision — Radford et al., arXiv
• WhisperX: Time-Accurate Speech Transcription of Long-Form Audio — Bain et al., arXiv
• WebVTT: The Web Video Text Tracks Format — W3C
• SubRip Subtitle format (SRT) — Library of Congress, Sustainability of Digital Formats
• EBU Tech 3293: EBUCore metadata set — European Broadcasting Union
• What is PBCore? — PBCore
• Features — PySceneDetect
• Preserving Audio, DPC Technology Watch Guidance Note (2021) — Digital Preservation Coalition and Artefactual Systems