Guide / RAG chunking strategies

RAG Chunking Strategies

Chunking decides what evidence retrieval can return. Bad chunks make good embedding models look broken.

What chunking controls

Chunking turns source documents into retrievable units. Each chunk should be small enough to rank precisely and large enough to answer the expected question.

The chunk is also the citation boundary. If a chunk mixes unrelated sections, retrieval and generation both become harder to debug.

Common chunking strategies

Fixed-size chunks are simple but can cut through headings, tables, and code blocks. Structure-aware chunks follow sections, markdown headings, functions, or tickets.

Overlap can preserve context across boundaries, but too much overlap inflates storage, duplicates results, and can crowd out diverse evidence.

Metadata is part of chunking

Store document type, product, version, language, tenant, source URL, heading path, and permissions with each chunk. Metadata lets retrieval filter before ranking and explain after ranking.

Complexity Table

Chunking strategy	Strength	Weakness	Best use
Fixed tokens	Simple and predictable	Breaks structure	Quick baselines
Heading-aware	Preserves document semantics	Needs parsing	Docs and knowledge bases
Code-aware	Keeps functions/classes intact	Language-specific	Developer docs and repositories
Small chunks plus parent context	Precise retrieval with broader answer context	More orchestration	Production RAG

When to Use This

Use structure-aware chunking when headings, tables, code blocks, or policy clauses carry meaning.
Use small retrieval chunks with parent expansion when precise matching and answer completeness both matter.

When Not to Use This

Do not use one global chunk size for every content type without measuring retrieval quality.
Do not add large overlap as a substitute for better parsing and metadata.

Production Failure Modes

Chunk boundary failures produce near-miss retrieval: the right document appears, but the necessary sentence is in a neighboring chunk.

Oversized chunks create diluted embeddings. The vector represents too many topics, so search retrieves broad pages instead of answerable evidence.

Animated SVG Diagram

A document splits into chunks with metadata, then selected chunks enter retrieval and reranking.

Next Topics

production RAG retrieval architecture RAG retrieval problems vector database memory calculator

FAQ

What is the best chunk size for RAG?

There is no universal best size. Start with your document structure and evaluate recall, answer quality, and citation usefulness on judged questions.

Should RAG chunks overlap?

Small overlap can help at boundaries, but excessive overlap increases storage and duplicate retrieval. Measure it.

Why does chunking affect vector search quality?

The embedding represents the chunk. If the chunk mixes topics or omits necessary context, nearest-neighbor search cannot recover the missing evidence.