← All resources

Chunking Strategies for RAG

Guide · updated 2026-08-05 · Markdown variant

Practitioner reference for chunking documents before embedding: fixed-size, recursive, semantic, late chunking, and contextual retrieval — with a strategy comparison table, chunk-size and overlap tradeoffs, code/table/Markdown handling, embedding model context limits, and evaluation methods.

Chunking is the decision that determines what the retriever can find. Every other RAG optimisation is downstream of it. This guide covers the five main strategies, the tooling that implements them, and how to choose. For the full retrieval pipeline see /resources/rag-retrieval-for-agents; for embedding model selection see /resources/embeddings-vector-search; for upstream parsing of PDFs and scans see /resources/document-extraction-for-agents.

Key facts

• Chunking decides what a retriever can find, so it's the upstream decision every other RAG optimization depends on.
• Recursive splitting (LangChain's RecursiveCharacterTextSplitter, LlamaIndex's SentenceSplitter) is the sensible default: it honors a document's existing structure, ingests quickly, and performs comparably to pricier semantic chunking across most production workloads — worth abandoning only once benchmarking proves another approach wins.
• Very short chunks (well under 100 tokens) score well on narrow-fact precision but strip away the surrounding context that would let a broader question actually match, hurting recall; very long chunks (well past 600 tokens) pack in more context per chunk at the cost of diluting the specific sentence a query needs and pulling the whole embedding toward a general topic instead.
• A practical starting point is roughly 256 to 512 tokens with 10 to 20 percent overlap, calibrated against precision@k and recall@k scores measured on a held-out set of queries; overlap only keeps boundary-straddling facts from vanishing, it doesn't fix a poor splitting strategy.
• Tables should never be split across chunks because a partial row carries no retrievable meaning, and code should be split on syntactic boundaries like functions and classes rather than by raw character count.
• Embedding models silently truncate text past their context window rather than erroring, so a chunk should stay at or below roughly 75% of the target model's limit — for example OpenAI's text-embedding-3 models cap at 8,191 tokens.

What strategy should I use?

Start with recursive splitting (LangChain RecursiveCharacterTextSplitter, LlamaIndex SentenceSplitter). It respects document structure, is fast at ingest time, and matches or beats semantic chunking in many production settings at far lower cost. Switch strategies only when evaluation shows a real gap.

Strategy comparison

Chunk size and overlap tradeoffs

There is no universal optimal chunk size — it depends on your query distribution and embedding model. Practical guidance:

• Too small (< ~100 tokens): high precision on narrow facts, but chunks lose the surrounding context that makes them meaningful; expect low recall on broader questions.
• Too large (> ~600 tokens): more context per chunk, but the relevant sentence is diluted by noise and the embedding drifts toward topic rather than specific fact.
• Typical starting points: 256–512 tokens with 10–20% overlap. Calibrate by measuring precision@k and recall@k on a held-out query set.
• Overlap prevents facts that straddle chunk boundaries from disappearing, but it does not rescue poor boundary placement — it is not a substitute for a better splitting strategy.

Chunking for code, tables, and Markdown

Code — split on syntactic boundaries (functions, classes), not character count. LangChain Language splitters use tree-sitter grammars; LlamaIndex CodeSplitter exposes chunk_lines and chunk_lines_overlap.

Tables — never split a table row across chunks; a partial row has no retrievable meaning. unstructured.io isolates Table elements as atomic chunks. For retrieval, convert tables to Markdown text or generate a natural-language summary alongside the raw table.

Markdown / HTML — use structural splitters that split on headings first. LangChain MarkdownHeaderTextSplitter splits on heading levels and preserves header metadata per chunk; LlamaIndex MarkdownNodeParser does the same. Header-as-metadata improves retrieval when queries reference section names.

Embedding model context limits and chunking

An embedding model silently truncates text that exceeds its context window, producing misleading embeddings. Known limits: OpenAI text-embedding-3-large/-small accept 8,191 tokens; jina-embeddings-v4 (current generation, released June 2025) natively supports up to 32,768 tokens, though Jina's hosted Embedding API currently caps input at 8,192 tokens regardless of model — check the live API docs before assuming the full native window is usable; the prior jina-embeddings-v3 accepts 8,192 tokens. Rule of thumb: keep chunks at or below ~75% of the embedding model's context limit to leave headroom and avoid edge-case truncation.

Tooling reference

LangChain (langchain-text-splitters): RecursiveCharacterTextSplitter (configure chunk_size, chunk_overlap), MarkdownHeaderTextSplitter (heading-aware, returns metadata), and Language splitters (tree-sitter, code-aware).

LlamaIndex (llama-index-core): SentenceSplitter (sentence-boundary-aware; chunk_size/chunk_overlap), SemanticSplitterNodeParser (buffer_size, breakpoint_percentile_threshold), and CodeSplitter (tree-sitter; language, chunk_lines).

unstructured.io (unstructured OSS): partition_* produces typed elements; chunk_by_title closes a chunk at each new section title (never merges across sections; max_characters, new_after_n_chars); Table elements stay atomic. Token-based chunking via max_tokens.

How to evaluate chunking choices

Chunking quality is best measured indirectly through retrieval metrics on your actual query distribution, not a generic benchmark.

1. Build a question set — for each source document, generate 5–10 questions whose answers live in specific passages (LLM-generated or real user queries).
2. Measure retrieval precision@k and recall@k — how many top-k retrieved chunks are relevant, and how many relevant chunks are in the top-k? Instrument with RAGAS (github.com/vibrantlabsai/ragas — rebranded from explodinggradients in 2026, old org link redirects; arXiv:2309.15217).
3. A/B compare strategies — hold retrieval and generation constant; vary only the chunking strategy and chunk size to isolate the effect.
4. Check for boundary failures — sample retrieved chunks; chunks starting/ending mid-sentence, mid-table-row, or mid-code-block indicate a poor splitter.

Verified sources

• LangChain RecursiveCharacterTextSplitter docs: https://python.langchain.com/docs/how_to/recursive_text_splitter/
• LangChain MarkdownHeaderTextSplitter docs: https://python.langchain.com/docs/how_to/markdown_header_metadata_splitter/
• LlamaIndex SentenceSplitter API reference: https://docs.llamaindex.ai/en/stable/api_reference/node_parsers/sentence_splitter/
• LlamaIndex semantic chunking example: https://developers.llamaindex.ai/python/examples/node_parsers/semantic_chunking/
• unstructured.io chunking docs: https://docs.unstructured.io/open-source/core-functionality/chunking
• Late Chunking paper (Jina AI / Weaviate, arXiv:2409.04701): https://arxiv.org/abs/2409.04701
• Contextual Retrieval (Anthropic, 2024): https://www.anthropic.com/engineering/contextual-retrieval
• OpenAI text-embedding-3-large model page: https://platform.openai.com/docs/models/text-embedding-3-large
• Jina Embeddings v4 announcement: https://jina.ai/news/jina-embeddings-v4-universal-embeddings-for-multimodal-multilingual-retrieval/
• jina-embeddings-v4 model page: https://jina.ai/models/jina-embeddings-v4/
• RAGAS framework (GitHub, vibrantlabsai; formerly explodinggradients): https://github.com/vibrantlabsai/ragas
• RAGAS paper (arXiv:2309.15217): https://arxiv.org/abs/2309.15217

#rag #chunking #embeddings #retrieval #langchain #llamaindex #agents #documents

Category: Guide