contenox

What is the KV cache? How much memory your context needs

Updated

UNDER THE HOOD What is the KV cache?

The KV cache is the memory a language model uses to remember the text it has already read. For every token in the context, every layer of the model stores two vectors, a key and a value, so it doesn’t have to recompute them for each new token it writes. It makes generation fast, and it is the reason a long context needs so much GPU memory: the cache grows with every token, on top of the model’s own weights.

What it looks like

A 30B coding model, quantized to 4 bits, downloads as about 19 GB. People see that number, buy or rent a 24 GB GPU, set the context to 256,000 tokens, and the model spills onto the CPU and crawls.

The missing piece is the KV cache. For that model, Qwen3-Coder-30B-A3B, it takes 96 KB per token at the default precision:

ContextKV cache
32,768 tokens3 GB
65,536 tokens6 GB
262,144 tokens24 GB

Weights and cache together have to fit. At 64k tokens that is about 25 GB before anything else runs on the card.

How does the KV cache work?

A model writes one token at a time. To choose the next token, each layer compares it against every earlier token, using a key and a value it computed for each of them. Without a cache, it would recompute all of those for the whole text on every step. With the cache, it computes them once, stores them, and only adds the new token’s key and value.

So the cache trades memory for speed, and the memory is proportional to the length of the context. It is also what prompt caching reuses: a provider that keeps the KV cache of your prompt’s beginning doesn’t have to read it again.

How much memory does the KV cache need?

bytes per token = 2 × layers × KV heads × head size × bytes per value
  • 2: one key and one value.
  • Layers, KV heads, head size: from the model’s config.json (num_hidden_layers, num_key_value_heads, head_dim). If head_dim is missing, it is hidden_size divided by num_attention_heads.
  • Bytes per value: 2 at the default f16 precision.

Multiply by the number of tokens in the context.

Worked through for Qwen3-Coder-30B-A3B (48 layers, 4 KV heads, head size 128):

2 × 48 × 4 × 128 × 2 bytes = 98,304 bytes = 96 KB per token
96 KB × 65,536 tokens = 6 GB

And for the smaller Qwen2.5-Coder-7B (28 layers, 4 KV heads, head size 128): 56 KB per token, 1.75 GB for its full 32,768-token context.

The KV-heads trap: use num_key_value_heads, not num_attention_heads. Most current models share each key and value across several attention heads (grouped-query attention). Qwen3-Coder-30B-A3B has 32 attention heads but only 4 KV heads. Use the wrong number and you calculate 48 GB for 64k tokens instead of 6 GB, and conclude you need a GPU you don’t.

We hit exactly this when we wrote the capacity planner for our own local inference daemon. It sizes the cache from KV heads, at the real byte width of the cache type, against a live reading of free GPU memory, because guessing either way gives you a crash or a context far smaller than the card could hold.

A calculator

Point it at any model on Hugging Face:

import json
import sys
import urllib.request

model = sys.argv[1]
tokens = int(sys.argv[2])
bytes_per_value = 2

url = f"https://huggingface.co/{model}/resolve/main/config.json"
with urllib.request.urlopen(url) as resp:
    c = json.load(resp)

layers = c["num_hidden_layers"]
kv_heads = c.get("num_key_value_heads", c["num_attention_heads"])
head_dim = c.get("head_dim") or c["hidden_size"] // c["num_attention_heads"]

per_token = 2 * layers * kv_heads * head_dim * bytes_per_value
total = per_token * tokens
print(f"{per_token / 1024:.0f} KB per token, {total / 1024**3:.2f} GB for {tokens} tokens")
python kv.py Qwen/Qwen3-Coder-30B-A3B-Instruct 65536
96 KB per token, 6.00 GB for 65536 tokens

Gated models need a Hugging Face token to read the config. Some architectures, such as models with sliding-window layers, need less than this formula says; treat it as the upper bound. Qwen3.5 and Qwen3.8 keep a full cache in only one layer out of four; best Ollama models for coding has the per-token figures for current local models.

How do I make the KV cache smaller?

  • Use less context. The cheapest fix. Agents need room, and Ollama’s docs recommend at least 64k tokens for coding tools, but 256k on a 24 GB card isn’t happening with this model.
  • Quantize the cache. Store keys and values in 8 bits instead of 16, roughly halving the memory. In Ollama, set OLLAMA_KV_CACHE_TYPE=q8_0. In llama.cpp, use --cache-type-k q8_0 --cache-type-v q8_0. Ollama’s docs describe the quality loss as very small and usually not noticeable.
  • Turn on flash attention, if your setup doesn’t already. It reduces memory use as the context grows. Ollama uses it automatically where supported; OLLAMA_FLASH_ATTENTION=1 forces it on.

The 8-bit caveat: “usually not noticeable” is true for chat. It is not the same as identical. When we saved and restored a session’s cache, f16 reproduced the model’s next tokens exactly, and the 8-bit cache measurably did not. If you need repeatable output, for tests, evaluations or comparing models, keep the cache at f16 if it fits.

How do I check what I actually got?

With Ollama, load the model and run:

ollama ps

CONTEXT shows the context length Ollama allocated. PROCESSOR shows whether everything fits on the GPU: anything other than 100% GPU means part of the model or cache spilled to the CPU, and it will be much slower. Lower the context or quantize the cache until it fits.

Related

Esc to close