Prompt caching
Providers cache the unchanged prefix of your prompt so repeated calls reprocess it at ~10% of the price and much lower latency.
In plain terms
If you hand a translator the same 50-page glossary before every sentence, they shouldn't re-read it each time. Prompt caching is the translator keeping the glossary open: the repeated part (system prompt, docs, tool definitions) is remembered; you pay full price only for what's new.
Why it matters
Real apps resend enormous identical prefixes constantly - long system prompts, RAG documents, conversation history, tool schemas. Without caching, you'd pay full input price for the same tokens thousands of times a day. Agents especially live or die by this.
How it works
Caching works on exact prefixes: everything must match byte-for-byte from the start of the prompt up to the cache point. So structure prompts stable-first: system prompt and tools (never change) -> documents (rarely change) -> conversation (grows at the end) -> newest message last. Anthropic uses explicit cache_control markers; OpenAI caches automatically past 1k tokens. Caches expire in minutes, refreshed on use.
When you use it
Long system prompts, chat history, agents (whole trajectory is resent every step), RAG with big documents, anything hitting the same context repeatedly.
Common mistakes
- Putting a timestamp or user name near the top of the system prompt - one changed byte kills the entire cache behind it.
- Reordering retrieved documents between calls, silently breaking prefix matching.
- Never checking the
cached_tokensusage field to confirm it's actually working.
Best practices
- Order prompt parts from most-stable to most-volatile.
- Verify cache hits in the usage response, not by faith.
- Design agents so the trajectory only appends, never rewrites.
Try it yourself
Send the same 5k-token document with two different questions, one minute apart. Compare cached_tokens and latency between call 1 and call 2.
Resources
- Anthropic - Prompt caching Cache breakpoints, pricing, TTLs.