Prompt caching lets a model provider reuse the work it already did on the beginning of your prompt. When a request starts with exactly the same text as a recent one, the provider skips reprocessing that part, answers faster, and bills those tokens at a fraction of the normal price. For AI agents this is not a detail: an agent resends its whole conversation on every step, so whether the cache hits decides most of the cost and much of the speed.
What it looks like
An agent an hour into a task. Every request it sends carries the whole conversation so far, often a few hundred thousand tokens. Without a cache hit, the model processes all of it again before it writes a word, on every one of the hundred or so requests behind each of your prompts. With a hit, it processes only the newest tool result.
The difference is easy to measure locally. In our own inference daemon, a stable prompt of about 9,000 tokens took 1.77 seconds to process cold and 0.03 seconds warm on a laptop GPU with 6 GB of memory. On a CPU, the same prompt took 7.61 seconds cold and 73 milliseconds warm. Editing the stable part of the prompt sent the next turn straight back to a full, cold prefill.
Hosted providers report it in every response. Anthropic’s usage block looks like this (illustrative numbers):
"usage": {
"cache_read_input_tokens": 78512,
"cache_creation_input_tokens": 1240,
"input_tokens": 36
}Most of the prompt came from the cache, a little was newly written to it, and only 36 tokens were processed at full price.
How does prompt caching work?
Before a model writes anything, it processes the entire prompt into an internal state, the KV cache. That step is called prefill, and for long prompts it is most of the work. Prompt caching keeps that state for a while, keyed on the exact beginning of the prompt.
The rule that matters: only an identical prefix is reused. The cache matches from the first token forward and stops at the first difference. Everything after a change is processed again, even if it is identical to last time.
That is why order matters. Anthropic’s docs spell out the order explicitly: tools, then the system prompt, then the messages. A change in the tool list invalidates everything after it.
How does each provider handle it?
| Provider | On by default? | How you control it | How you see hits |
|---|---|---|---|
| Anthropic | No: you ask for it | A top-level cache_control for automatic caching, or cache_control on up to 4 content blocks | cache_read_input_tokens and cache_creation_input_tokens |
| OpenAI | Yes, on supported models | Optional prompt_cache_key to influence routing | cached_tokens in the usage details |
| Gemini | Yes, implicit caching on Gemini 2.5 and newer | Explicit caching on supported Gemini 3 and 2.5 models | total_cached_tokens in the usage |
| llama.cpp server | Yes, cache_prompt defaults to true | --cache-reuse N to also reuse shifted chunks | Server timings |
| vLLM | Check your version | --enable-prefix-caching | Server metrics |
A few rules that are easy to miss, from the providers’ own docs:
- Anthropic keeps a cache entry for 5 minutes by default, refreshed for free every time it is used. A 1-hour lifetime costs extra. Writing to the cache costs a bit more than normal input; reading costs a small fraction. Prompts below a minimum length, between 512 and 4,096 tokens depending on the model, are never cached, and no error tells you so.
- OpenAI caches automatically once a prompt passes the model’s minimum length. Its docs say the whole rendered prefix must match.
- Gemini applies implicit caching automatically above a minimum length: 2,048 tokens on Gemini 2.5 Flash and Pro, 4,096 on the newer models.
Prices and minimums change. Check the Anthropic, OpenAI and Gemini pages before you plan around them.
What breaks the cache in an agent session?
Each of these changes something early in the prompt, so everything after it is processed again at full price:
- A timestamp in the system prompt. “Today is 4 November, 14:32” changes every request. Put dates at the end, or leave out the time.
- A tool list that changes. An MCP server connecting halfway through a session, tools sorted differently, or a tool description that includes a counter. Tools come first, so this invalidates everything.
- Retrieved text placed early. RAG passages differ with every question. Put them after the stable part. This is a big reason coding agents stopped embedding repositories: see What are embeddings?
- Editing history. Rewriting or summarizing earlier messages changes the prefix. Compacting a conversation is worth it, but expect the next turn to pay full price once.
- Pauses. On Anthropic’s default lifetime, stepping away for more than five minutes means the next turn rebuilds the cache. For agents that wait on a human, the 1-hour option can pay for itself.
- Switching models or routes. Caches belong to one model, and usually one provider deployment. Switching models mid-session, or a router sending you to a different backend, starts cold.
The silent-miss trap: a cache miss never shows up as an error. The answer is just as good, only slower and more expensive. The only way to know is to read the usage numbers. In Claude Code,
/usageshows cache reads and writes for the session.
How do I keep the cache warm?
- Stable first, changing last. Tools, then instructions, then project context, then the conversation, with the newest content at the very end.
- Append, don’t edit. Treat the conversation as a log you only add to.
- Keep the tool list fixed for a session. Decide which tools and MCP servers an agent has before it starts.
- Keep one model per session. Use a different model for a different task, in a new session.
- Watch the numbers for a week. If cache reads aren’t the large majority of your input tokens in long sessions, something in the list above is happening.
What about local models?
Locally there is no bill, but prefill time is the whole story: on consumer hardware it can take seconds per turn, as the numbers above show. The same rule applies: only an identical prefix is reused.
- llama.cpp’s server compares each prompt with the previous one in the same slot and only processes the new part, by default. Its docs warn that this can make results not bit-for-bit identical between runs.
- vLLM reuses shared prefixes across requests with
--enable-prefix-caching. Its docs note it only speeds up prefill, not token generation.
vLLM’s docs warn that newer Claude Code builds send a per-request attribution header that can defeat prefix caching when Claude Code talks to a local vLLM server. vLLM versions above 0.17.1 handle it automatically. On older versions, set CLAUDE_CODE_ATTRIBUTION_HEADER to "0" in the env section of ~/.claude/settings.json. Nothing errors; every turn is just slow.
When we built our own local inference daemon, we went further: the stable part of an agent’s prompt was hashed, and that hash predicted reliably whether the next turn would hit the cache. Reuse was keyed on more than the text, too. The model file, the prompt template and the runtime settings all had to match, because the same text through a different template is a different prompt.
When prompt caching doesn’t help
- Short prompts. Below the provider’s minimum length, nothing is cached.
- Long answers. Caching speeds up reading the prompt, not writing the answer. A response that generates thousands of tokens is still slow.
Related guides
- AI coding costs: where agent tokens go, and other ways to cut them
- What are embeddings?: why retrieval and caching don’t mix well
- How to run local LLMs for coding: context length and the local setup
- What is MCP?: tools that load mid-session and what they cost