Phase 05 · Embeddings & vector search

NLP Pipeline Architecture

The sequence of steps that turns raw text into something a model can actually use.

In plain terms

Every text-based AI system runs the same journey underneath: raw messy text goes in, structured meaning comes out. The NLP pipeline is the sequence of stages that makes that conversion reliable. Understanding it is what separates people who call an API from people who can debug why the answers are wrong.

Why it matters

Most real text is messy - PDFs, scans, HTML, inconsistent encodings. The pipeline is where that becomes usable input, and where most retrieval quality is won or lost.

How it works

Work through the stages in order: acquire text, clean and normalize it, segment it, then represent it numerically. Skip the stages your task does not need rather than running all of them by habit.

When you use it

Any time you are processing documents before embedding, indexing, or classifying them.

Common mistakes

  • Running every stage out of habit when the task needs only two of them.
  • Ignoring extraction quality - garbage from a bad PDF parse poisons everything downstream.
  • Chunking before cleaning, so boilerplate ends up inside your embeddings.

Best practices

  • Decide which stages you need by asking what the downstream task requires.
  • Inspect the text after each stage before moving to the next.
  • Keep the raw source alongside the processed text so you can re-run the pipeline.

Try it yourself

Take one messy PDF, run it through extraction and cleaning, and diff the raw text against the cleaned text to see exactly what your pipeline removed.

Deep dive

Every text-based AI system runs the same journey underneath: raw messy text goes in, structured meaning comes out. The NLP pipeline is the sequence of stages that makes that conversion reliable. Understanding it is what separates people who call an API from people who can debug why the answers are wrong.

NLP pipeline architecture diagram showing six stages: data ingestion from documents, APIs, databases, emails, logs and audio; text cleaning and normalization; segmentation and tokenization; linguistic and analytical processing; representation through sparse, static and transformer embeddings stored in a vector database; and a detection layer covering classification, named entity recognition, sentiment analysis, summarization, question answering, translation and text generation.
The full NLP pipeline: six stages that turn raw text into structured intelligence, with the output of each stage feeding the next. Open full size ->

01

Data Ingestion

Collect raw text from wherever it actually lives. This stage is unglamorous and it is where most projects quietly lose quality, because a bad extraction cannot be repaired downstream.

  • Documents (PDF, DOCX, HTML)
  • Web pages and APIs
  • Databases and data warehouses
  • Emails, tickets, and chat logs
  • Audio and speech transcripts

Output Raw text data

02

Text Cleaning and Normalization

Standardize the surface form of the text so that two identical meanings do not look like two different inputs. Every choice here is a trade-off: aggressive cleaning speeds up classical models but can destroy signal that transformers would have used.

  • Lowercasing and Unicode normalization
  • HTML, URL, and boilerplate removal
  • Punctuation and whitespace handling
  • Number normalization
  • Stopword handling
  • Spelling correction

Output Clean text

03

Segmentation and Tokenization

Split continuous text into the units a model can consume. Modern LLMs use subword tokenization, so a token is not a word: it is a frequent character sequence learned from a corpus. This is why token counts and billing rarely match your word count.

  • Sentence segmentation
  • Word tokenization
  • Subword tokenization (BPE, WordPiece, SentencePiece)
  • Chunking by paragraph or sliding window
  • Document structuring

Output Tokens and chunks

04

Linguistic and Analytical Processing

Extract explicit grammatical and semantic structure. This stage is optional and task-dependent: transformer models learn much of it implicitly, but you still want it when you need interpretable features, rule-based logic, or work in a low-resource language.

  • Stemming and lemmatization
  • Morphological analysis
  • Part-of-speech tagging
  • Dependency and constituency parsing
  • Chunking into noun and verb phrases
  • Coreference resolution

Output Linguistic features

05

Representation (Feature Extraction)

Convert text into numbers. The representation you choose decides what the system can and cannot understand, and it is the single highest-leverage decision in the pipeline.

  • Sparse: one-hot encoding, bag of words, TF-IDF
  • Static dense: Word2Vec, GloVe, FastText
  • Contextual: BERT, RoBERTa, sentence transformers, LLM embeddings
  • Storage: vector databases such as FAISS, Chroma, Pinecone, or Weaviate

Output Vectors in an embedding store

06

Detection and Intelligence Layer

The stage users actually see. Everything above exists to make this layer accurate. The same prepared representation can serve several of these tasks at once.

  • Text classification and sentiment analysis
  • Named entity recognition and information extraction
  • Summarization
  • Question answering
  • Machine translation
  • Text generation

Output Structured responses, insights, and actions

What it is

An NLP pipeline is an ordered set of transformations between unstructured human language and a machine-usable representation. Each stage has a defined input and a defined output, which is what makes the whole thing debuggable: when results are wrong, you can walk backwards through the stages and find the one that broke.

Why it still matters in the LLM era

It is tempting to assume large language models removed the need for a pipeline. They did not; they absorbed part of it. Stages 03 and 05 now happen inside the model, and stage 04 is often implicit. But ingestion, cleaning, and chunking remain entirely yours, and they are where most production quality problems originate. A model cannot recover meaning that your PDF extractor already destroyed.

How it maps to a RAG pipeline

A RAG system is this pipeline with retrieval bolted between representation and generation. Ingestion, cleaning, chunking, and embedding are stages 01 to 05 exactly as shown. Retrieval queries the vector store built in stage 05, and generation is one task inside stage 06. If your RAG app returns irrelevant chunks, the fault is almost always in stages 02 and 03, not in the model.

Choosing what to skip

Not every project needs every stage. A sentiment classifier on clean product reviews can go from stage 02 straight to a transformer embedding. A legal search system over scanned contracts will spend most of its effort on stages 01 and 02 and barely touch stage 04. Decide by asking what the downstream task actually needs, then delete the rest rather than running it out of habit.

Common mistakes

  • Applying classical cleaning such as stopword removal and stemming before a transformer model, which strips the exact context the model relies on.
  • Chunking by fixed character count so sentences and tables are cut in half.
  • Evaluating only the final output, so you never learn which stage introduced the error.
  • Treating tokens as words when estimating cost and context limits.
  • Using a different preprocessing path at query time than the one used at indexing time.

Best practices

  • Log the intermediate output of every stage for a sample of inputs, and read them yourself.
  • Keep indexing-time and query-time preprocessing in one shared code path.
  • Chunk on structural boundaries such as headings, paragraphs, or sentences before falling back to length.
  • Match cleaning to the model: aggressive for TF-IDF and classical models, minimal for transformers.
  • Version your pipeline. Changing the chunker or the embedding model invalidates the whole index and requires a re-embed.

Practical workflow

raw text -> clean and normalize -> tokenize and chunk -> (optional linguistic features) -> embed -> store vectors -> task model -> structured output

FAQ

What are the stages of an NLP pipeline?

Data ingestion, text cleaning and normalization, segmentation and tokenization, linguistic and analytical processing, representation or feature extraction, and the detection layer that performs the actual task such as classification, NER, summarization, or generation.

Do large language models replace the NLP pipeline?

No. LLMs absorb tokenization, representation, and most linguistic analysis, but ingestion, cleaning, and chunking are still your responsibility and still determine output quality.

What is the difference between tokenization and chunking?

Tokenization splits text into the smallest units a model consumes, usually subwords. Chunking groups text into larger passages, usually paragraphs or sliding windows, so a retrieval system can store and return coherent context.

Should I remove stopwords before using BERT or an LLM?

Generally no. Stopword removal helps sparse models such as TF-IDF, but transformers use those words to resolve grammar and reference, so removing them usually lowers accuracy.

Which tools are used to build an NLP pipeline?

spaCy and NLTK for classical processing, Hugging Face Transformers for models and tokenizers, scikit-learn for sparse representations and classical classifiers, and FAISS, Chroma, or Pinecone for storing embeddings.

Related topics

Chunking strategies Splitting documents into right-sized pieces before embedding - the humble step that quietly decides retrieval quality more than any model choice. What are embeddings An embedding turns text into a list of numbers (a vector) positioned so that similar meanings land close together in space. The RAG pipeline, end to end Wiring ingestion and query-time retrieval into one system, where overall quality is the product of every stage's quality.