An embedding is a list of numbers that an AI model produces to represent the meaning of a piece of content, arranged so that items with similar meaning get similar numbers. Vector search uses those numbers to find the items closest in meaning to a query, rather than the items that share its exact words.
This is the search technology behind many AI assistants that answer questions from documents, and increasingly behind archive search, related-content suggestions and duplicate detection. It is powerful, but it is not a replacement for keyword search, and it has failure modes that content teams should understand before relying on it.
This guide explains how embeddings capture similarity, how vector databases search large collections quickly, why keyword search with BM25 still matters, how hybrid search combines the two, how multilingual embeddings and metadata filters work, and how to decide which approach a search problem needs.
Key takeaways
- Embeddings place content with similar meaning close together, so vector search can match a query to relevant material even when they share no words.
- Vector databases use approximate nearest-neighbour indexes such as HNSW to search large collections quickly, trading a small amount of accuracy for speed.
- Keyword search with BM25 remains a strong baseline and is usually better for exact names, codes and rare terms.
- Hybrid search, which merges keyword and vector results, is a sensible default for most content archives.
- No single embedding model is best at everything, so test candidates on your own content and real queries.
What an embedding is
Think of a map. Every town has coordinates, and towns that are near each other on the map are near each other in reality. An embedding model does something similar for meaning: it reads a sentence, paragraph, image or audio clip and outputs coordinates, typically several hundred or a few thousand numbers long, in a space where related content sits close together.
No single number means anything a person could name. What matters is position relative to everything else. A paragraph about a hospital's waiting lists will sit near other paragraphs about health service delays, even if one says 'waiting list' and another says 'backlog of patients'. A paragraph about football transfers will sit far away from both.
The models learn these positions by training on very large amounts of text, adjusted so that passages people treat as related end up near each other. Sentence-BERT (Reimers and Gurevych, 2019) showed why separate embeddings matter for search. Comparing sentences with the original BERT model meant feeding every pair through the model together: finding the most similar pair among 10,000 sentences took about 50 million model computations, roughly 65 hours. Sentence-BERT encodes each sentence once into a fixed vector, and the same task took about 5 seconds. That principle is what makes vector search work at archive scale: you embed every document once, then embed each query once and compare.
Similarity, intuitively
Picture each embedding as an arrow pointing from the centre of the map. Cosine similarity, a common comparison, asks how closely two arrows point in the same direction, ignoring how long they are. Arrows pointing exactly the same way score 1; unrelated content scores lower. Other measures, such as Euclidean distance (straight-line distance between the arrow tips) and inner product, are also used, and databases such as pgvector support all three.
Two cautions follow. First, similarity scores are relative: a score that signals a strong match for one model may be ordinary for another, so do not carry thresholds from one model to the next. Second, similar is not the same as relevant. 'The council approved the plan' and 'the council rejected the plan' are about the same topic and may sit close together. Embeddings tend to capture what a passage is about better than they capture negation, specific numbers or exact identifiers.
How vector databases search millions of items
The simplest vector search compares the query with every stored vector and returns the closest. This exact search has perfect recall, but its cost grows with the size of the collection. For large archives, vector databases use approximate nearest-neighbour (ANN) indexes, which find almost all of the true closest matches in a fraction of the time.
HNSW
A widely used ANN method is the Hierarchical Navigable Small World graph, described by Yury Malkov and Dmitry Yashunin in a paper first released in 2016. It links each item to its near neighbours in a stack of graph layers. Every item appears in the bottom layer; a random, exponentially shrinking share also appears in each layer above, so the top layers are sparse.
A road network is a good analogy. To reach a street in another city, you take the motorway to the right region, then main roads, then local streets. HNSW search starts in the sparse top layer, moves greedily towards the query, then drops down a layer and repeats with finer steps until it reaches the bottom. The authors showed this gives logarithmic complexity scaling, which is why search stays fast as collections grow.
HNSW has settings that trade speed for recall. In pgvector, the PostgreSQL extension, they are m (maximum connections per item in each layer, default 16), ef_construction (effort during building, default 64) and hnsw.ef_search (candidates considered per query, default 40). Higher values find more of the true neighbours but run more slowly.
| Index type | Recall | Query speed | Build cost | Notes |
|---|---|---|---|---|
| Exact (no index) | Perfect | Slows as the collection grows | None | Fine for small collections or narrow filters |
| HNSW | High, tunable | Fast | Slower to build, more memory | Can be built before data is loaded (pgvector) |
| IVFFlat | Lower for the same speed | Fast | Faster to build, less memory | Divides vectors into lists; build after data is loaded for good recall |
Keyword search and BM25 still matter
Keyword search finds documents that contain the query's words. A widely used ranking function, BM25, scores each match using three ideas:
- Rare words count more. A word that appears in few documents is a stronger signal than one that appears in almost all of them (inverse document frequency).
- Repetition helps, up to a point. A document mentioning a term five times scores higher than one mentioning it once, but the benefit levels off. The
k1parameter controls how quickly; Elasticsearch's default is 1.2. - Long documents are adjusted. A match in a short document counts for more than the same match in a very long one. The
bparameter controls this; the default is 0.75.
BM25 is old, cheap and transparent, and it is hard to beat on some problems. The BEIR benchmark (Thakur et al., NeurIPS 2021) tested ten retrieval approaches on 18 datasets without training them on those datasets. It found BM25 to be a robust baseline across domains, while dense (embedding-based) models often underperformed, leaving considerable room to improve how well they generalise; re-ranking and late-interaction models scored best on average, at higher computational cost. Embedding models have changed a great deal since 2021, but the lesson holds: on content unlike its training data, a vector model can lose to keyword search, so test on your own material.
Hybrid search: combining both
Hybrid search runs keyword search and vector search on the same query and merges the two result lists. Each catches what the other misses.
Take a generic example. A user of a regional newspaper archive searches for 'A&E waiting times at St Mary's'. Vector search returns articles about hospital waiting times in general, some about other hospitals. Keyword search finds every article mentioning 'St Mary's', including ones about a school and a church. Merged, the results that rank well in both lists, articles about waiting times at that hospital, rise to the top.
A simple, popular merging method is reciprocal rank fusion (RRF), described by Cormack, Clarke and Büttcher at SIGIR 2009. Each document gets a score from each list based only on its rank: one divided by (a constant plus its rank), with the constant set to 60 in the paper. The scores are added up. A document ranked second in both lists beats one ranked first in one list but missing from the other. Because RRF uses ranks rather than raw scores, it does not need keyword and vector scores to be on the same scale. In the authors' experiments it outperformed the Condorcet fusion method and the best individual system by 4–5% on average.
Some embedding models produce keyword-style and vector outputs together. M3-Embedding (Chen et al., 2024), for example, supports dense, sparse and multi-vector retrieval from a single model, covers more than 100 languages and accepts inputs of up to 8,192 tokens. Hybrid results are often then passed to a re-ranking model that reads the query and each candidate together; the pgvector documentation points to examples of both RRF and cross-encoder re-ranking. Hybrid search is a core part of retrieval-augmented generation.
Multilingual embeddings and metadata filters
Searching across languages
Multilingual embedding models are trained so that the same meaning lands in roughly the same place whatever the language. A query in English can then retrieve a relevant passage in French or Polish without translating either. Quality is not uniform: models are usually strongest in languages with abundant training data. If your archive includes smaller languages, test retrieval in each one rather than relying on an average score. The guide to multilingual archives covers language detection, normalisation and translation.
Filtering by metadata
Most real searches combine meaning with hard constraints: articles from 2015–2019, only the business section, only resources for a given class level, only content this user may see. Vector databases store metadata alongside each vector for this purpose, but how filters interact with ANN indexes matters.
The pgvector documentation gives a clear example. With approximate indexes, filtering is applied after the index is scanned. If a condition matches 10% of rows and HNSW returns its default 40 candidates, only about 4 rows will match on average, so a request for the top 10 may return fewer results than expected. Remedies include iterative index scans (available from pgvector 0.8.0), which keep scanning until enough matches are found; partial indexes or partitions for common filter values; and exact search when a filter is narrow enough. Other vector databases handle filtering differently, so check how yours does it before relying on filters with a small number of matches.
Filters are only as good as the metadata behind them. Inconsistent dates, missing section labels and free-text subject fields quietly break filtered search.
When to use vector search, keyword search or both
| Search need | Better fit | Why |
|---|---|---|
| Natural-language questions | Vector or hybrid | Matches meaning, not wording |
| Synonyms and paraphrase | Vector | 'Backlog of patients' finds 'waiting lists' |
| Exact names, ISBNs, case numbers, product codes | Keyword | Needs an exact match on a precise token |
| New or rare terms the model may not have seen | Keyword | No reliance on what the model learned in training |
| Queries in one language, content in another | Multilingual vector | Shared meaning space across languages |
| Date ranges, sections, access permissions | Metadata filters | A constraint, not a similarity question |
| Explaining why a result matched | Keyword | Matched terms can be highlighted |
| Finding near-duplicates and related items | Vector | Similar content clusters together |
Vector search is also useful well beyond answering questions. Embeddings power related-story suggestions, clustering for taxonomy work and near-duplicate detection.
Choosing and testing an embedding model
The MTEB benchmark (Muennighoff et al., 2022) evaluated 33 models across 8 task types, 58 datasets and 112 languages and found that no single embedding method dominated across all tasks. Public leaderboards are a starting point, not an answer. Before choosing:
- Collect 100–200 real queries from search logs or editors, with the documents that should be found for each.
- Embed a representative sample of your archive with two or three candidate models.
- Compare how often the right document appears in the top 5 or 10 results, for keyword, vector and hybrid search.
- Check language coverage, maximum input length and the cost of embedding the full collection.
- Plan for change: vectors from different models are not comparable, so switching models means re-embedding everything.
How Druid Learning helps
Vector and hybrid search can only be as good as the text and metadata they index. Druid Learning is an AI archival processing platform, based in Dublin and New York, that turns content archives into AI-ready datasets. It handles content in 140+ languages and formats including OCR scans, PDF, XML, EPUB, MP3 and MP4, and Archive Optimization turns an archive in any format into an indexed, labelled, AI-ready dataset. For NPR, 2M+ assets were processed with 20+ labels across 8 dimensions, including topic, location and article type, supporting archive search by label alongside suggested tags and related stories for journalists.
Frequently asked questions
What is the difference between an embedding and a vector?
A vector is simply an ordered list of numbers. An embedding is a vector produced by a trained model to represent the meaning of a piece of content, such as a sentence, paragraph, image or audio clip. All embeddings are vectors, but not all vectors are embeddings. In search, the two words are often used interchangeably, as in 'vector search' and 'embedding search'.
Do I need a dedicated vector database?
Not necessarily. Extensions such as pgvector add vector indexes, including HNSW, to PostgreSQL, so organisations already using a relational database can start there. Dedicated vector databases may offer more options for very large collections, filtering and scaling. The more important decisions are usually which embedding model to use, how to chunk content and how to combine vector results with keyword search.
Is vector search better than keyword search?
Neither is better in general. Vector search handles natural-language questions, synonyms and paraphrase well, while keyword search with BM25 is more reliable for exact names, identifiers and rare terms and is easier to explain. Benchmarks have found BM25 to be a robust baseline across domains. For most content archives, hybrid search that merges both result lists gives the best overall results.
Can embeddings work across languages?
Yes. Multilingual embedding models are trained to place text with the same meaning close together regardless of language, so a query in one language can retrieve relevant content written in another. Some current models cover more than 100 languages. Quality varies by language, though, and is usually lower for languages with little training data, so test retrieval separately for each language in your archive.
Do I have to re-embed my archive if I change models?
Yes. Each embedding model produces its own space, and vectors from different models cannot be meaningfully compared, even when they have the same number of dimensions. Switching models means re-embedding every document and every query. Budget for this when choosing a model, and keep the source text and chunking settings so the collection can be regenerated consistently.
Sources
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks — Reimers and Gurevych, EMNLP 2019 (arXiv)
- Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs — Malkov and Yashunin (arXiv)
- pgvector: Open-Source Vector Similarity Search for Postgres — pgvector (GitHub)
- Practical BM25, Part 2: The BM25 Algorithm and its Variables — Elastic
- BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models — Thakur et al., NeurIPS 2021 (arXiv)
- Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods — Cormack, Clarke and Büttcher, SIGIR 2009 (University of Waterloo)
- M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation — Chen et al. (arXiv)
- MTEB: Massive Text Embedding Benchmark — Muennighoff et al. (arXiv)