Phase 05 · Embeddings & vector search Core

What are embeddings

An embedding turns text into a list of numbers (a vector) positioned so that similar meanings land close together in space.

In plain terms

Imagine a magic map where every sentence gets GPS coordinates based on meaning: 'I love pizza' and 'pizza is amazing' land meters apart; 'my car broke down' lands in another city. Once meaning is a location, 'find similar text' becomes 'find nearby points' - something computers do brilliantly.

Why it matters

Keyword search fails at synonyms: search 'refund' and miss the ticket saying 'I want my money back'. Embeddings solve the vocabulary-mismatch problem - matching by meaning, not spelling. They are the foundation of semantic search, recommendations, clustering, deduplication and RAG retrieval.

How it works

A trained embedding model reads text and outputs a fixed-length vector (commonly 384-3072 numbers). Each dimension isn't human-labeled; collectively they encode topic, tone, intent. Closeness is measured with cosine similarity (angle between vectors): ~1 = same meaning, ~0 = unrelated. You embed once, store the vectors, and compare cheaply forever.

When you use it

Search over your own data, 'related items', grouping similar feedback, detecting duplicates, routing queries by topic - and always as step one of RAG.

In code

import numpy as np, openai
client = openai.OpenAI()

def embed(text):
    r = client.embeddings.create(model="text-embedding-3-small", input=text)
    return np.array(r.data[0].embedding)

a, b = embed("a kitten rested on the rug"), embed("the cat sat on the mat")
c = embed("quarterly revenue grew 12%")
cos = lambda x, y: x @ y / (np.linalg.norm(x) * np.linalg.norm(y))
print(cos(a, b))  # ~0.75 - similar meaning
print(cos(a, c))  # ~0.05 - unrelated

Common mistakes

  • Comparing vectors from two different embedding models - coordinates from different maps are meaningless together; re-embed everything when you switch models.
  • Using an LLM for similarity when embeddings do it 1000x cheaper.
  • Embedding whole 50-page documents as one vector - meaning gets averaged into mush (-> chunking).

Best practices

  • Cache embeddings - same text, same model, same vector; never pay twice.
  • Store the source text and metadata alongside each vector; you'll always need it back.
  • Normalize obvious noise (boilerplate headers/footers) before embedding.

Try it yourself

Embed 'the cat sat on the mat', 'a kitten rested on the rug' and 'quarterly revenue grew 12%'. Compute pairwise cosine similarity. The numbers will make the whole concept click.

Resources

Related topics

Similarity metrics The math for 'how close are two vectors' - cosine similarity, dot product, Euclidean distance - and it matters less than you fear. Embedding models Dedicated models (not chat LLMs) that produce embeddings - you choose by quality-per-dollar, dimension size, and language support. Chunking strategies Splitting documents into right-sized pieces before embedding - the humble step that quietly decides retrieval quality more than any model choice. Build: semantic search end-to-end The rite-of-passage build: ingest documents -> chunk -> embed -> index -> query - the complete retrieval pipeline that RAG will sit on. The vocabulary: 15 terms you'll see everywhere A one-line dictionary of the words the whole field assumes you already know. The NLP pipeline, end to end The sequence of steps that turns raw text into something a model can actually use.