Phase 05 · Embeddings & vector search Core

Chunking strategies

Splitting documents into right-sized pieces before embedding - the humble step that quietly decides retrieval quality more than any model choice.

In plain terms

You can't ask a librarian to hand you 'the relevant book' when your answer lives in one paragraph. Chunking is pre-cutting books into passages so search can return the exact paragraph. Cut too small and answers lose their surroundings; too big and each piece is a blurry average of many topics.

Why it matters

Embedding a whole document averages all its meanings into one point - a contract's payment terms and termination clause collapse together. Chunking preserves precision, and it's empirically the highest-leverage knob in most retrieval systems.

How it works

Baseline that works: split by structure (headings, paragraphs) into ~300-800 token chunks with ~10-15% overlap so sentences straddling a boundary survive. Better: respect semantic units - never split mid-table, keep code blocks whole, keep a Q&A pair together. Attach metadata to every chunk (source, title, section, page) and consider prepending a breadcrumb ('Doc: Employment Contract > Section 4: Termination') so chunks carry their own context.

When you use it

Every RAG/search ingestion pipeline. Revisit whenever retrieval returns technically-similar-but-unhelpful fragments.

Common mistakes

  • Fixed 500-character splits that cut sentences and tables in half (the default in old tutorials).
  • No overlap, so the key sentence sits half in chunk 12 and half in chunk 13, matching neither.
  • Losing metadata, so you can retrieve a paragraph but can't tell the user which document or page it came from.
  • One chunking policy for wildly different content (contracts, code, chat logs each need their own).

Best practices

  • Look at your chunks! Print 20 random ones - if a human can't tell what each is about, neither can the embedding.
  • Chunk by document structure first, size second.
  • Store chunks so you can also fetch their neighbors ('small-to-big' retrieval: match small, feed the model the surrounding section).

Try it yourself

Chunk the same manual three ways: 200 tokens, 600 tokens with overlap, and by section headings. Run the same 5 queries against each index and rank which chunking wins per query.

Resources

Related topics

What are embeddings An embedding turns text into a list of numbers (a vector) positioned so that similar meanings land close together in space. The RAG pipeline, end to end Wiring ingestion and query-time retrieval into one system, where overall quality is the product of every stage's quality. Context window The maximum number of tokens a model can consider at once - its entire working memory, including your prompt, the history, and its answer. Tokens & tokenization Models don't read words or letters - they read tokens, chunks of ~4 characters, and every cost, limit and quirk traces back to them. The NLP pipeline, end to end The sequence of steps that turns raw text into something a model can actually use. Vector databases Databases purpose-built to store millions of embeddings and answer 'find the K nearest vectors to this one' in milliseconds. Build: semantic search end-to-end The rite-of-passage build: ingest documents -> chunk -> embed -> index -> query - the complete retrieval pipeline that RAG will sit on. Documents, loaders and text splitters A Document is text plus Metadata, usually created by a Document Loader and broken into Chunk objects by a Text Splitter.