Linear Attention Explainer
Linear attention refers to methods that reduce the sequence-length cost of standard self-attention. Ordinary attention forms interactions between every query and every key, creating quadratic time or memory growth.
Many linear methods replace the softmax similarity with feature maps that allow matrix multiplication to be reassociated. Instead of building the full attention matrix, the model summarizes keys and values first and then combines that summary with each query.
The approach can support longer text, vision, audio, and streaming sequences with lower memory. Some variants use random features or recurrent formulations and can generate tokens with a fixed-size state.
Approximating or changing softmax attention can reduce accuracy or alter what patterns are easy to learn. Numerical stability and causal masking also matter. Evaluation should compare quality, speed, memory, hardware utilization, and sequence lengths actually used.
Linear attention rearranges or approximates attention calculations so cost grows roughly linearly rather than quadratically with sequence length.
Key terms include linear attention, self-attention, quadratic cost, feature map, kernel, query, key, value, causal mask and recurrence.
Keys and values are summarized before each query reads from them.
This changes sequence-length growth from pairwise interactions toward linear processing.
A linear approximation may not preserve every property of softmax attention.
Benchmarks should pair long-context speed with accuracy and stability.
Explore more "Explainers"
Discover additional explainers across politics, science, business, technology, and other fields. Each explainer breaks down a complex idea into clear, everyday language—helping you better understand how major concepts, systems, and debates shape the world around us.
