Phase 05 · Embeddings & vector search Core

Vector databases

Databases purpose-built to store millions of embeddings and answer 'find the K nearest vectors to this one' in milliseconds.

In plain terms

Comparing your query against 10 million vectors one-by-one is like finding a face in a stadium by checking every seat. Vector databases build clever shortcuts (indexes) that check only the promising sections - thousands of times faster, at the price of very rarely missing the absolute best seat.

Why it matters

Exact nearest-neighbor search is O(N) per query and melts at scale. Approximate nearest neighbor (ANN) indexes make semantic search over huge corpora fast and cheap enough for production. Everything RAG retrieves flows through one of these.

How it works

They index vectors with structures like HNSW (a navigable graph of neighborhoods - the common default). You upsert {id, vector, metadata, text}, then query with a vector + optional metadata filters ('only docs where team=legal') to get top-K matches with scores. Options tier by ops effort: libraries (FAISS - in-process, no server), embedded/local (Chroma, LanceDB - pip install and go), servers (Qdrant, Weaviate, Milvus), managed (Pinecone), or your existing Postgres with pgvector - often the sanest production choice.

When you use it

Roughly past 50k-100k vectors, when you need metadata filtering at scale, or multi-user production apps. Below that, numpy or Chroma is genuinely fine - don't add infrastructure for a demo.

Common mistakes

  • Choosing database infrastructure before having 100 users - classic premature optimization; start with Chroma/pgvector.
  • Ignoring metadata filtering, then hacking access control after retrieval (filter AT query time).
  • Never updating the index, so the bot confidently cites deleted docs.
  • Believing vendor benchmark wars - for most apps, all mainstream options are fast enough.

Best practices

  • Store text + rich metadata with every vector, and plan an update/delete story from day one.
  • If you already run Postgres, try pgvector before adding a new database to your stack.
  • Keep an 'index version' so you can rebuild embeddings without downtime.

Try it yourself

Install Chroma. Ingest 100 paragraphs from Wikipedia articles with {topic} metadata. Run 5 semantic queries, then the same queries with a metadata filter - feel both halves of the query model.

Resources

  • Chroma quickstart A working local vector DB in 5 minutes.
  • pgvector Vector search inside Postgres - the boring, excellent choice.

Related topics

Embedding models Dedicated models (not chat LLMs) that produce embeddings - you choose by quality-per-dollar, dimension size, and language support. Chunking strategies Splitting documents into right-sized pieces before embedding - the humble step that quietly decides retrieval quality more than any model choice. The RAG pipeline, end to end Wiring ingestion and query-time retrieval into one system, where overall quality is the product of every stage's quality. Hybrid search & reranking Combine semantic search with old-school keyword search (hybrid), then have a smarter model re-order the candidates (reranking) - the two standard upgrades when plain RAG plateaus. Similarity metrics The math for 'how close are two vectors' - cosine similarity, dot product, Euclidean distance - and it matters less than you fear. Build: semantic search end-to-end The rite-of-passage build: ingest documents -> chunk -> embed -> index -> query - the complete retrieval pipeline that RAG will sit on. Agent memory Memory is selected context from prior turns or external stores that helps an agent continue work. Retrievers and RAG chains A Retriever turns a query into relevant Source Documents using embeddings, vector search and ranking strategies.