What Is a KV Cache?

KV Cache Explainer

A key-value cache, or KV cache, stores the attention keys and values computed for earlier tokens during autoregressive generation. When the model produces the next token, it creates only the new key and value and reuses the cached history.

Without the cache, the model would repeatedly recompute attention representations for the entire prefix. Reuse sharply reduces computation, but the cache grows with sequence length, layer count, batch size, and the number and width of key-value heads.

The KV cache is central to high-throughput language-model serving. Systems allocate cache blocks, move or evict them, share common prefixes, quantize entries, and use attention variants that store fewer heads.

Long contexts and many simultaneous requests can exhaust GPU memory even when model weights fit. Fragmentation and preemption also waste capacity. Evaluation should track time to first token, per-token latency, throughput, memory, and eviction behavior.

A KV cache stores attention keys and values from earlier tokens so a language model does not recompute them for every new token.

Key terms include KV cache, key, value, attention, autoregressive generation, prefix, GPU memory, cache block, eviction and preemption.

Keys and values for the existing prefix stay available while new tokens are generated.

Only the newest token’s entries need to be added at each step.

Cache memory grows across layers, tokens, requests, and key-value heads.

Block allocation, quantization, sharing, and eviction determine serving capacity.

Explore more "Explainers"

Discover additional explainers across politics, science, business, technology, and other fields. Each explainer breaks down a complex idea into clear, everyday language—helping you better understand how major concepts, systems, and debates shape the world around us.