Documents, loaders and text splitters
A Document is text plus Metadata, usually created by a Document Loader and broken into Chunk objects by a Text Splitter.
In plain terms
Before retrieval works, your files must become clean pieces of text. PyPDFLoader loads PDFs, WebBaseLoader loads pages, CSVLoader loads rows, DirectoryLoader walks a folder, and RecursiveCharacterTextSplitter creates chunks with a chosen Chunk Size and Chunk Overlap.
Why it matters
Bad ingestion creates bad RAG. If the PDF text is mangled or chunks cut important context apart, no model or agent can recover reliably.
How it works
Load source files into Document objects, preserve Metadata like source path and page number, split into Chunks, tune Chunk Size and Chunk Overlap, then inspect samples before embedding.
When you use it
Any document QA, knowledge base, retrieval agent, citation feature or long-context processing system.
Common mistakes
- Indexing extracted text without reading it first.
- Using one chunk size for tables, prose and code.
- Dropping metadata, then being unable to cite Source Documents later.
Best practices
- Inspect loaded Documents before splitting.
- Keep source and page metadata through every stage.
- Tune chunking with a retrieval eval, not vibes.
Try it yourself
Load one PDF with PyPDFLoader, one webpage with WebBaseLoader, one CSV with CSVLoader and a notes folder with DirectoryLoader, then split them with RecursiveCharacterTextSplitter.