Phase 02 · How LLMs actually work

Cost & pricing mechanics

You pay per million tokens, input and output priced separately - and architecture decisions, not usage volume, dominate the bill.

In plain terms

Model pricing is like international calling: a rate per minute (token), different for talking (output, expensive) vs listening (input, cheaper). A chatbot that resends the whole conversation every turn is paying for the same 'minutes' again and again.

Why it matters

AI apps die of cost more often than of quality. A feature that costs $0.002 per use is a business; the same feature at $0.30 per use might not be. Engineers who can estimate cost on a napkin make better designs.

How it works

Cost = (input tokens x input rate) + (output tokens x output rate). Example: 100k requests/day, 2k in + 500 out each, on a $3/$15-per-million model -> (0.2B x $3 + 0.05B x $15)/1M ~= $1,350/day. Levers: smaller models, shorter prompts, prompt caching (-> 90% off repeated prefixes), batch APIs (~50% off non-urgent work), capping max_tokens.

When you use it

Do the napkin math BEFORE building any feature, and again before scaling it.

Common mistakes

  • Forgetting output tokens cost 3-5x input tokens.
  • Not noticing conversation history makes cost grow quadratically over a chat session.
  • No per-user/per-feature cost tracking until the surprise invoice arrives.

Best practices

  • Log the usage field of every response into your analytics from day one.
  • Set billing alerts and hard limits on every provider account.
  • Estimate cost-per-user-per-month for any feature before you pitch it.

Try it yourself

Napkin exercise: price a customer-support bot for 1,000 conversations/day, 10 turns each, 300 tokens per turn, on two different models. Which parts of the math shock you?

Resources

Related topics

Tokens & tokenization Models don't read words or letters - they read tokens, chunks of ~4 characters, and every cost, limit and quirk traces back to them. Prompt caching Providers cache the unchanged prefix of your prompt so repeated calls reprocess it at ~10% of the price and much lower latency. Choosing the right model Picking a model is balancing four axes - quality, cost, latency, context size - for your specific task, not finding 'the best model'. Latency and cost engineering The craft of making AI responses faster and cheaper without ruining quality.