Multi-Query Attention Explainer
Multi-query attention is a transformer attention variant in which many query heads share a single key head and a single value head. Standard multi-head attention gives every head separate query, key, and value projections.
During autoregressive generation, past keys and values are stored in a cache. Sharing them greatly reduces cache size and the memory bandwidth needed to read that cache for each new token.
The method can improve decoding throughput and support larger batches or longer contexts on the same hardware. Query heads still learn different ways to read from the shared key-value representation.
Sharing removes capacity from the key and value side and may reduce quality for some models or tasks. Conversion and training choices matter. Comparisons should include perplexity or task quality, latency, throughput, cache memory, and batch size.
Multi-query attention gives each attention head its own queries while all heads share one set of keys and values.
Key terms include multi-query attention, query head, key, value, shared projection, KV cache, memory bandwidth, decoding and throughput.
Each query head remains distinct, while keys and values are shared across all heads.
The smaller cache reduces memory reads during token-by-token generation.
One key-value representation must serve every query head.
Quality and speed tradeoffs should be measured at realistic batch and context sizes.
Explore more "Explainers"
Discover additional explainers across politics, science, business, technology, and other fields. Each explainer breaks down a complex idea into clear, everyday language—helping you better understand how major concepts, systems, and debates shape the world around us.
