AI Engineering · interactive
The KV cache
To pick each new token, attention needs a key and a value for every token before it. Keep them, and each step only does work for the newest token. Throw them away, and every step starts over. Switch between the two and watch the counters.
Prefill
with KV cache
The capital of France is Paris
| token | key (K) | value (V) | this step |
|---|---|---|---|
| The | computed | ||
| ·capital | computed | ||
| ·of | computed | ||
| ·France | computed | ||
| ·is | computed |
Rows computed so far, no cache5
Rows computed so far, with cache5
Prefill: the model reads all 5 prompt tokens at once, computes a key and a value for each, and produces “Paris”.
1 The prompt tokens are processed together, in parallel. Each gets a key and a value, and they're stored. This is why the first token takes longest.
5 rows computed, the same with or without a cache
2 3 4 5
In production
Tools and technologies
Every serving stack keeps a KV cache; the differences are in how they store and share it.
Not tied to one language — use these names when you search docs or design a system.
- vLLMSoftware
PagedAttention stores the KV cache in pages to fit more users per GPU.
GitHub ↗ - Hugging Face TransformersSoftware
Cache classes for generation, including quantized and offloaded caches.
Documentation ↗ - llama.cppSoftware
Local inference with a configurable context size and cache type.
GitHub ↗ - Prompt cachingService
Providers reuse the cached prefix of a repeated prompt and charge less for it.
Documentation ↗