Latency and cost engineering
The craft of making AI responses faster and cheaper without ruining quality.
In plain terms
A demo that takes 20 seconds and costs too much is not a product. Smaller models, fewer tokens, caching, and parallel calls are the usual fixes.
Why it matters
Tokens dominate both price and latency, so cost work and speed work often point to the same improvements.
How it works
Measure tokens in, tokens out, time to first token, total latency, and cost per request. Then reduce prompt size, cap output, cache repeated work, and route easy tasks to cheaper models.
When you use it
After the feature works and before real traffic grows.
Common mistakes
- Optimizing model choice before trimming a bloated prompt.
- Only measuring average latency.
- Caching user-specific or time-sensitive answers too broadly.
Best practices
- Put cost and latency beside quality metrics.
- Re-run evals after every optimization.
- Set spend caps in code.
Try it yourself
Log 20 requests, calculate cost per request, apply a token diet, and compare the before and after table.