Inside the model: attention, in 10 minutes
The transformer architecture processes all tokens in parallel and uses 'attention' to let every token look at every other token and decide what matters.
In plain terms
In 'The animal didn't cross the street because it was tired', how do you know 'it' means the animal? You glanced back and weighed the options. Attention is exactly that glance, done with math, by every token toward every other token, many times over.
Why it matters
Older models (RNNs) read text one word at a time and forgot the start of long sentences; they also couldn't be parallelized, so they couldn't scale. The 2017 'Attention Is All You Need' paper fixed both - that unlock is why LLMs got big enough to be useful.
How it works
Tokens become vectors (lists of numbers). Each attention layer lets every token gather information from the tokens most relevant to it, then a small neural network transforms the result; stack ~30-100 such layers and the final vector for the last position becomes a probability score for every possible next token. Pick one, append it, repeat - that's generation, one token at a time (why responses 'type out' gradually).
When you use it
You never implement this - but the intuition explains real behavior: why generation is sequential (latency), why long contexts cost so much (attention is pairwise), why position in context matters.
Common mistakes
- Going down a 6-month deep-learning-math rabbit hole before building anything - as an AI engineer, intuition is required, calculus is optional.
- Skipping it entirely and treating the model as pure magic - you'll misdiagnose latency and context problems forever.
Best practices
- Watch one great visual explanation (below), be able to sketch it on a napkin, move on.
- Revisit after a month of building - it will click twice as hard.
Try it yourself
Napkin test: draw text -> tokens -> vectors -> [attention layers] -> probabilities for next token -> pick -> repeat. If you can label the arrows, you know enough.
Resources
- 3Blue1Brown - Attention in transformers, visually The clearest attention visualization on the internet.
- Jay Alammar - The Illustrated Transformer The classic diagram-first walkthrough.