Phase 02 · How LLMs actually work Core

Context window

The maximum number of tokens a model can consider at once - its entire working memory, including your prompt, the history, and its answer.

In plain terms

Picture a whiteboard of fixed size. Everything the model 'knows' about your conversation must fit on that whiteboard; the model has no memory besides it. When the board fills, something must be erased - that's why long chats 'forget' the beginning.

Why it matters

Attention (the mechanism inside transformers) compares every token with every other token - costs grow steeply with length, so a hard limit exists. The context window is the boundary you engineer around; RAG, summarization and memory systems all exist because of it.

How it works

Modern windows: ~128k tokens (GPT-4o), 200k (Claude), 1M+ (Gemini). Sounds huge, but a 300-page book is ~150k tokens and long agent sessions burn context fast. Also: models attend unevenly - information buried in the middle of a huge context is recalled worse ('lost in the middle'), and every token in context is paid for on every single call.

When you use it

Every design decision about 'how much do I show the model': conversation history, retrieved documents, tool results, code files.

Common mistakes

  • Stuffing everything into context because it fits - cost and accuracy both degrade; relevance beats volume.
  • Letting chat history grow unbounded until requests crawl and bills explode.
  • Assuming the model weighs all context equally - put critical instructions at the start and end.

Best practices

  • Send the least sufficient context: retrieve relevant pieces (-> RAG) instead of dumping raw files.
  • Summarize or trim old conversation turns past a threshold.
  • Track context size per request in logs; it's your main cost lever.

Try it yourself

Paste a long article into a chatbot and ask about a detail from the middle paragraph. Then ask again with only that paragraph. Compare precision - you've just discovered why RAG exists.

Resources

Related topics

Tokens & tokenization Models don't read words or letters - they read tokens, chunks of ~4 characters, and every cost, limit and quirk traces back to them. What is RAG & why it exists RAG = retrieve relevant documents for the user's question, paste them into the prompt, and instruct the model to answer only from them. Prompt caching Providers cache the unchanged prefix of your prompt so repeated calls reprocess it at ~10% of the price and much lower latency. Agent memory Memory is selected context from prior turns or external stores that helps an agent continue work. Prompt engineering vs context engineering Prompt engineering crafts the instructions; context engineering builds the systems that decide what information fills the model's window at runtime. Chunking strategies Splitting documents into right-sized pieces before embedding - the humble step that quietly decides retrieval quality more than any model choice.