Phase 02 · How LLMs actually work Core

Tokens & tokenization

Models don't read words or letters - they read tokens, chunks of ~4 characters, and every cost, limit and quirk traces back to them.

In plain terms

'Understanding' might be one token, but 'antidisestablishmentarianism' becomes six. Think LEGO bricks for text: common words are one brick; rare words get built from several. You pay per brick, and the model can only hold so many bricks at once.

Why it matters

Computers need numbers, not letters. Tokenization is the compromise between one-number-per-character (sequences too long) and one-per-word (vocabulary too huge). Every LLM limitation you'll hit - price, context limits, even why models are bad at counting letters - is a token story.

How it works

A tokenizer has a fixed vocabulary (~100k chunks) learned from data. Your text is greedily split into the longest matching chunks, each mapped to an ID: "AI is fun" -> [15837, 374, 2523]. The model only ever sees those IDs. Rule of thumb: 1 token ~= 4 characters ~= ¾ of an English word; other languages often cost 2-3x more tokens.

When you use it

Whenever you estimate cost, hit a context limit, truncate documents for RAG, or wonder why the model can't spell 'strawberry' backwards (it can't see letters).

Common mistakes

  • Estimating cost by word count instead of token count.
  • Chunking documents by characters and accidentally splitting mid-token/mid-sentence.
  • Forgetting you pay for BOTH input and output tokens - long conversations resend everything, every turn.

Best practices

  • Play with a live tokenizer until it's intuitive.
  • Log token usage (usage field in every API response) from your very first app.
  • When output is cut off mid-sentence, check max_tokens before blaming the model.

Try it yourself

Open OpenAI's tokenizer playground and paste: your name, some Python code, an emoji, and a Hindi sentence. Predict the token count before each - you'll be wrong in instructive ways.

Resources

Related topics

Context window The maximum number of tokens a model can consider at once - its entire working memory, including your prompt, the history, and its answer. Cost & pricing mechanics You pay per million tokens, input and output priced separately - and architecture decisions, not usage volume, dominate the bill. Chunking strategies Splitting documents into right-sized pieces before embedding - the humble step that quietly decides retrieval quality more than any model choice. The vocabulary: 15 terms you'll see everywhere A one-line dictionary of the words the whole field assumes you already know. What is an LLM A large language model is a program trained on huge amounts of text to predict the next token - and that one trick, scaled up, produces reasoning, coding and conversation. Inside the model: attention, in 10 minutes The transformer architecture processes all tokens in parallel and uses 'attention' to let every token look at every other token and decide what matters.