KV Cache (Key-Value Cache)
The KV cache stores the key and value vectors the attention mechanism has already computed for every token in the context, so the model never recomputes them. It is what makes generation after the first token fast — and it is a memory cost that scales linearly with context length, independently of the model's weights.
Why it grows
Each layer caches one key and one value vector per token. Cache size is:
kv_bytes = 2 × n_layers × d_model × context_length × batch × bytes_per_element
A 70B model with 80 layers at 8k context in FP16 needs several gigabytes of cache on top of the weights; push to 128k context and the cache alone can exceed the weights.
Levers
| Technique | Effect | Cost |
|---|---|---|
| FP8 / INT8 KV cache | Halves or quarters cache size | Small quality loss at long context |
| Grouped-query attention (GQA) | Fewer KV heads → smaller cache | Built into the model architecture |
| PagedAttention (vLLM) | No fragmentation, higher batch | Serving-framework dependent |
| Sliding-window / eviction | Bounded cache | Loses information outside the window |
The practical takeaway: when you calculate whether a model fits in VRAM, the KV cache is the term that blows up your budget as you raise context length or concurrency — not the weights.
END OF ANALYSIS