Chunking Long Documents for AI Retrieval: Strategies and Trade-offs

Chunking is the step in an AI retrieval system where long documents are split into smaller passages that can be searched, ranked and passed to a language model individually. How you split a document decides what the system can find: cut in the wrong place and the answer to a question ends up divided between two chunks, neither of which ranks well.

There is no universally correct chunk size, but there is useful evidence about what works. This guide compares fixed-size, structure-aware and semantic chunking, explains the trade-offs of chunk size and overlap, shows how to keep metadata and parent-document context with each chunk, covers tables, lists, PDFs and scanned pages, and sets out how to test chunking choices on your own content.

Key takeaways

• Structure-aware chunking that follows headings and paragraphs, with a maximum size as a safety limit, is the best starting point for most archives.
• Semantic chunking can help, but one study found its extra computational cost was not justified by consistent gains, so test it before adopting it.
• Smaller chunks retrieve more precisely and larger ones carry more context; parent-document retrieval lets a system use both.
• Every chunk should carry its document ID, title, heading path and key metadata so it can be filtered, cited and understood on its own.
• Popular default settings can perform poorly, so evaluate chunking with real questions from your own collection.

Why chunking matters

Retrieval systems compare a query with stored passages, usually by converting both into embeddings (see embeddings and vector search). An embedding of a 300-page book blurs hundreds of topics into one point and matches everything weakly. An embedding of one sentence is precise but may lack the context to be recognised as relevant. Chunking chooses the unit in between.

The choice has always been part of retrieval-augmented generation. The original 2020 RAG paper by Lewis et al. split each Wikipedia article into disjoint 100-word chunks, about 21 million in all. Since then, the evidence has shown the choice matters. In a 2024 technical report, Chroma researchers Brandon Smith and Anton Troynikov found that some chunking strategies outperformed others by up to 9% in recall, and that 'default settings for certain popular chunking strategies can lead to relatively poor performance'.

A good chunk has three properties:

1. It makes sense on its own. A reader shown only this chunk could tell what it is about.
2. It is about one thing. It does not mix two unrelated topics that would each match different queries.
3. It does not cut through meaning. Sentences, list items, table rows and short sections stay whole.

Three families of chunking strategy

Fixed-size chunking

The text is cut every N tokens, words or characters, regardless of content. It is simple, fast and gives predictable sizes, which makes cost and context budgets easy to plan. The drawback is that boundaries fall wherever the count runs out: mid-sentence, between a heading and its first paragraph, or halfway through a table. It works reasonably for uniform running text such as transcripts or novels, and poorly for structured documents.

Structure-aware chunking

The text is split along boundaries the author already created. The LangChain documentation describes two kinds. Text-structure splitting works down the natural hierarchy of writing: its recursive splitter tries to keep paragraphs intact, falls back to sentences if a paragraph is too long, and goes down to words only if necessary. Document-structure splitting follows the format itself, such as Markdown headers, HTML tags, JSON objects or functions in code. LlamaIndex offers equivalent parsers for Markdown, HTML and JSON.

For articles, reports, textbooks and manuals, splitting by section and then by paragraph, with a maximum size, keeps related content together. This is the approach to try first.

Semantic chunking

Instead of using structure or counts, semantic chunkers look at meaning. A common method, implemented in LlamaIndex's semantic splitter, embeds each sentence (or small group of sentences) and starts a new chunk where similarity between neighbours drops sharply, for example above the 95th percentile of distances. The LlamaIndex documentation notes that its sentence splitting is mainly suited to English and that the threshold may need tuning.

The evidence is mixed. Qu, Tu and Bao (2024) compared semantic chunking with simple fixed-size chunking on document retrieval, evidence retrieval and answer generation and concluded that its computational costs 'are not justified by consistent performance gains'. In the Chroma report, an LLM-based chunker and an embedding-based clustering chunker achieved the two highest recall scores, but the authors noted the LLM chunker could take tens of minutes to run.

A related idea is to index propositions: short, self-contained statements that each express a single fact. Chen et al. (Dense X Retrieval) trained a model to rewrite passages into propositions and found that proposition-level indexing outperformed passage-level retrieval and improved question answering for a fixed amount of retrieved text.

Chunk size and overlap

Chunk size is a trade-off. Smaller chunks give more precise matches and waste less of the model's context on irrelevant text; larger chunks keep more surrounding explanation and are less likely to separate a question's answer from its context.

The Chroma report gives some useful reference points. Its evaluation used token-level recall, precision and intersection-over-union (IoU), which penalises retrieving irrelevant or repeated text, across five corpora and 472 queries. With OpenAI's text-embedding-3-large model and five chunks retrieved:

• A recursive splitter at 200 or 400 tokens with no overlap performed well; the authors described the 200-token setting as consistently high performing across all metrics.
• 800-token chunks with 400 tokens of overlap, which the report identifies as the default for OpenAI's Assistants file search, tested with a token-based splitter, had slightly below-average recall and the lowest scores on the other measures.
• The authors found that reducing overlap improved IoU, because overlapping chunks retrieve the same text twice.

The results also depended on the embedding model. With a smaller open-source model (all-MiniLM-L6-v2), overlap helped: a token-based splitter with 250-token chunks and 125 tokens of overlap reached 82.4% recall, compared with 77.1% without overlap.

Practical guidance follows from this:

• Start with structure-aware chunks of roughly 200–500 tokens, merging very short paragraphs and splitting very long ones.
• Use little or no overlap when splitting on paragraph boundaries; consider some overlap when cutting fixed windows through running text.
• Never exceed the embedding model's maximum input length, or text beyond the limit may be ignored.
• Treat any number, including these, as a starting point for your own tests rather than a rule.

Keeping metadata and parent context with each chunk

A chunk removed from its document loses information a reader would take for granted: which publication, which date, which chapter, who 'she' refers to. Put that information back.

Store metadata on every chunk

Each chunk should carry a stable reference to its source and the fields people will filter or cite by. A generic example:

{"chunk_id": "doc-48213#s3-p2", "document_id": "doc-48213", "title": "Council approves new housing plan", "heading_path": "Local news > Housing", "published": "2024-03-18", "language": "en", "page": 4, "position": 7, "text": "The council voted to approve the plan after a two-hour debate..."}

Prepend context to the text you embed

Adding the document title and heading path to the start of each chunk's text before embedding gives the model context the chunk itself lacks. Anthropic's 'contextual retrieval' method goes further, using a language model to write a short, chunk-specific explanation (typically 50–100 tokens) that places each chunk within its document. In Anthropic's 2024 evaluation, this reduced the rate of failed top-20 retrievals by 35%, by 49% when combined with keyword (BM25) search on the same contextualised text, and by 67% with re-ranking added. Generating context for every chunk has a cost, so weigh it against the size of your collection.

Retrieve small, read large

Parent-document approaches index small units for precise matching but give the language model a larger surrounding unit. LlamaIndex documents two versions. Its sentence-window parser indexes single sentences and stores a window of neighbouring sentences (three on each side in its example) in metadata, swapping the window in before the model reads it. Its hierarchical parser creates chunks at several sizes (2,048, 512 and 128 tokens in the example), each linked to its parent; an auto-merging retriever replaces child chunks with their parent when most of a parent's children are retrieved.

Tables, lists and other awkward structures

Generic splitters handle running prose well and almost everything else badly.

• Tables. Keep a table in one chunk if it fits. If not, split by rows and repeat the header row in every piece. Store tables as Markdown or HTML with their title and caption; a table flattened into unlabelled numbers is effectively lost.
• Lists. Keep the introductory sentence with its items. A list of steps without its lead-in ('To apply for the grant:') answers nothing.
• Captions, sidebars and boxes. Attach captions to their figures, and treat sidebars and boxed features as separate chunks so they do not interrupt the main text.
• Repeated page furniture. Remove running headers, footers and page numbers from the text, but keep page numbers in metadata for citations.
• Entries. Glossaries, dictionaries, FAQs and question banks should be chunked one entry per chunk.
• Transcripts. Chunk by speaker turns or time windows and keep timestamps in metadata so answers can link to the right moment.

PDFs and scanned pages

PDF is a page-description format: it records where characters appear on a page, and many PDFs carry little or no information about the order in which text should be read. Naive extraction can interleave columns, merge headers and footers into body text, split words hyphenated across lines and flatten tables. Chunking that output reproduces every error. The guide to publishing file formats explains what each format does and does not store.

The safer pattern is to convert first and chunk second:

1. Recover structure. Use a layout-aware converter that identifies headings, paragraphs, tables, figures and reading order. Docling, an open-source toolkit, is one example: it uses a layout analysis model and a table-structure model called TableFormer.
2. OCR scanned pages. Pages that are images need OCR before anything else, and OCR errors carry straight into retrieval. See OCR for archives for accuracy checks.
3. Export to a structured format. Markdown, HTML or JSON with headings and tables marked up lets structure-aware chunkers do their job.
4. Keep page references. Record source page numbers on every chunk so citations can point to the original page.
5. Spot-check. Look at a sample of chunks from difficult pages (multi-column layouts, tables, footnotes) before indexing a whole collection.

Evaluating chunking choices

Choose chunking by testing rather than by convention:

1. Build a question set. Collect real questions from users or editors and mark the passage, ideally the exact span, that answers each. Language models can help generate questions, as in the Chroma study, but its authors noted that generated questions are stylistically narrow, so mix in real ones.
2. Hold everything else constant. Compare chunking strategies with the same embedding model, retrieval settings and number of results.
3. Measure retrieval. Check how often the answering passage appears in the top results (recall at k). Token-level metrics, as used by Chroma, also show how much irrelevant text each strategy retrieves.
4. Measure answers. Run end-to-end tests to check whether generated answers are correct and supported by the retrieved chunks.
5. Read the failures. Look at cases where retrieval missed. They usually point to a specific cause: a table split from its header, a heading separated from its section, a chunk that only makes sense with the previous one.
6. Re-test after changes. A new embedding model or new content types can change which strategy works best.

How Druid Learning helps

Chunking works best on content whose structure has been recovered and labelled before it is split. Druid Learning is an AI archival processing platform, based in Dublin and New York, that turns content archives into AI-ready datasets. It processes formats including OCR scans, PDF, XML, IDML, INDD, SVG, EPUB, WAV, MP3 and MP4, and Archive Optimization turns an archive in any format into an indexed, labelled, AI-ready dataset. For CJ Fallon, an Irish educational publisher, 50,000 resources were processed and mapped to the curriculum, the kind of label that can travel with every chunk as filterable metadata.

Frequently asked questions

What is the best chunk size for RAG?

There is no single best size; it depends on the content, the embedding model and the questions asked. As a starting point, structure-aware chunks of roughly 200–500 tokens work well for many collections, and one published evaluation found 200–400-token recursive chunks with no overlap performed strongly. Test two or three sizes on real questions from your own archive before committing.

Should chunks overlap?

Sometimes. Overlap repeats text at the end of one chunk at the start of the next, which reduces the chance that an answer is cut in half when boundaries fall arbitrarily. It also means the same text can be retrieved twice, wasting context. When chunks follow paragraph or section boundaries, little or no overlap is usually needed; for fixed windows through running text, modest overlap can help.

Is semantic chunking better than fixed-size chunking?

Not reliably. Semantic chunking splits text where its meaning shifts, which can help with long unstructured text. However, a 2024 study found its computational cost was not justified by consistent performance gains over fixed-size chunking, while another evaluation found some semantic methods performed well. Where documents have headings and paragraphs, structure-aware chunking is usually a simpler and stronger starting point.

How should tables be chunked for AI retrieval?

Keep each table in a single chunk if it fits within the size limit, stored in a structured format such as Markdown or HTML with its title and caption. If a table is too large, split it by rows and repeat the header row in every chunk so each piece still makes sense on its own. Avoid flattening tables into unlabelled text.

What is parent-document retrieval?

Parent-document retrieval indexes small chunks, such as single sentences or short paragraphs, for precise matching, but passes a larger surrounding unit, such as the full section, to the language model when one of those chunks is retrieved. It combines the precision of small chunks with the context of large ones. Frameworks such as LlamaIndex provide sentence-window and hierarchical versions.

Sources

• Evaluating Chunking Strategies for Retrieval — Chroma Technical Report
• Is Semantic Chunking Worth the Computational Cost? — Qu, Tu and Bao (arXiv)
• Dense X Retrieval: What Retrieval Granularity Should We Use? — Chen et al. (arXiv)
• Text Splitter Integrations — LangChain Docs
• Node Parser Modules — LlamaIndex Docs
• Introducing Contextual Retrieval — Anthropic
• Docling Technical Report — Auer et al. (arXiv)
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al., NeurIPS 2020 (arXiv)