← All guides · RAG in production

How to Chunk Documents for RAG (Strategy Beats Size)

Part 2 of RAG in production · 938 words · ~4 min read · published 2026-08-23 · updated 2026-08-23 · Markdown variant

Chunking decisions that actually move retrieval quality: structural boundaries before fixed windows, parent-document expansion, special handling for tables and code, overlap trade-offs, and tuning against recall@k instead of blog defaults.

Part of the Agentic RAG in Production: The Complete Operator Guide guide.

Chunking decides retrieval quality before any model participates, because embedding and ranking operate on whatever units you hand them. The pillar guide frames chunking as following query shape; this article gives the working decision procedure. The recurring anti-pattern is starting from a number — "~512 tokens" — copied from a quickstart and never validated. The number is a parameter; the strategy is the decision.

Which chunking strategy should you use?

Work down this list and stop when a strategy fits your content:

1. Structural splitting — for anything with headings, sections, list items or table rows. Structure is free coherence signal: the boundaries already mark complete thoughts.
2. Recursive or semantic splitting — for unstructured prose: split on paragraph and sentence boundaries, then merge until near the target size. Respects sentence integrity where raw windows do not.
3. Parent-document retrieval — when queries need precision but answers need context: index small units, expand each hit to its enclosing section at assembly time.
4. Proposition-level chunking — for dense reference material where single facts must be retrievable in isolation, accepting more storage and more index entries.

Fixed-window slicing remains the fallback for genuinely unstructured text — and even there, sentence-aligned windows beat character-aligned ones. The deeper taxonomy of splitters and their failure modes lives in the chunking strategies reference; this page stays at the decision level.

How do tables and code change the rules?

Tables sliced mid-row destroy the row-to-header relationships that made them informative. Keep tables whole together with their captions, store a linearized text rendering alongside any visual extraction, and split oversized tables by logical groups of rows with headers repeated per fragment — never by raw token count.

Code wants semantic boundaries: whole functions or classes, with imports and module context attached. A function split from its signature, or a call site split from its definition, embeds poorly and matches worse. Configuration files chunk naturally per resource or stanza.

How should you chunk content that updates in place?

Corpora are not frozen: policies gain paragraphs, API docs gain endpoints, wikis get edited. Chunk boundaries interact with edits because a small insertion can shift every downstream boundary under naive re-splitting, which changes chunk IDs and forces wholesale re-embedding of an untouched document. Alignment-aware splitters — anchored to headings or explicit markers rather than raw offsets — localize the blast radius of an edit so only genuinely affected chunks change identity. Whatever scheme you pick, verify its edit behavior directly: insert one sentence into a sample document and count how many chunks changed. That number is your incremental-update cost, and it belongs in the strategy decision alongside recall@k.

What does overlap actually buy?

Overlap keeps boundary-spanning ideas retrievable from at least one chunk. Its costs scale past storage: duplicated passages create near-duplicate hits that crowd result lists and demand deduplication downstream. Practical guidance:

• Start near 10–15% of chunk size
• Reduce overlap under parent-document retrieval — the parent supplies continuity
• Re-check overlap whenever average document structure changes

Treat overlap like any retrieval knob: adjusted by measurement, not by anxiety.

How do chunking choices interact with embedding models?

Chunk boundaries define what each embedding must represent, so the two choices are coupled. Very large chunks force one vector to average multiple ideas, blurring its location in embedding space; very small chunks produce vectors that match sharply but carry too little context to disambiguate homonyms and pronouns. Models also impose a maximum input length — chunks sized near that ceiling risk truncation, which silently discards content at exactly the boundary you chose. Check the interaction empirically: embed a sample under candidate configurations and compare retrieval metrics rather than assuming independence.

Multilingual corpora add one more constraint: mixed-language chunks dilute both lexical statistics and embedding semantics. Where a document mixes languages at natural boundaries (sections, paragraphs), align chunk boundaries to the language switch; where it interleaves within sentences, keep language as metadata so filters can scope queries later.

When should you re-chunk an existing index?

Three triggers justify the cost: measured retrieval decay on the golden set that traces to chunk-boundary failures; a corpus whose document mix shifted materially (new document class, new language, new structure); and an embedding-model migration, which already forces a rebuild and makes it the cheap moment to change splitters too. Absent those triggers, leave working chunking alone — churn without a metric behind it spends reindex budget to move sideways.

How do you tune chunking without guessing?

Hold a golden set of real questions annotated with the passages that must be retrieved. For each candidate configuration, rebuild an index over a representative sample and measure recall@k plus MRR on the set. Compare candidates on those numbers; ship the winner; keep the loser's numbers in the notes so future regressions have context. Chunk-size debates without a retrieval metric attached are aesthetics (evaluating AI agents covers the harness itself).

Two operational cautions. First, changing chunking re-tokens and re-indexes everything — schedule it like a migration with a dual-index window, exactly as the embeddings article describes for model changes. Second, log the chunking config version alongside every index so an index can always be traced back to the exact splitter parameters that produced it; unexplained mixed-vintage indexes are a common source of "retrieval got weird last week" mysteries.

A final note on defaults: they are scaffolding, not decisions. The fastest teams treat their first chunking configuration as a hypothesis with a measurement plan attached, replace it the moment better numbers appear, and record every configuration's metrics in the eval notes so the history of what was tried survives personnel changes.

#rag #chunking #retrieval #production