CPU Cache Explainer
A CPU cache is a small, fast memory that keeps copies of instructions and data close to the processor’s execution units. Main memory can hold far more information, but reaching it takes many processor cycles. When a core requests an address, cache hardware checks whether the corresponding block is already nearby. A cache hit supplies the bytes quickly. A miss sends the request to a lower cache level or to DRAM, and the returning block is placed in the cache for possible reuse. This automatic process hides much of the memory delay.
Caches work because programs show locality. Temporal locality means recently used data is likely to be used again, as with a loop counter or repeatedly called instructions. Spatial locality means nearby addresses are likely to be accessed soon, as with consecutive array elements. Hardware moves memory in fixed-size cache lines rather than individual bytes. Reading one value therefore brings neighboring values with it. Sequential work benefits, while scattered access can fetch many lines whose unused bytes consume bandwidth and capacity.
Modern processors use a hierarchy. Level-one instruction and data caches are small and extremely fast, usually private to a core. A larger level-two cache catches many L1 misses at somewhat higher latency. A last-level cache is larger again and is often shared among cores, though exact organization varies. Each level trades speed, capacity, power, and chip area. The lookup may happen in parallel with address translation and prediction so a hit can feed the pipeline without a long pause.
Every line needs a tag identifying which memory address it represents and state describing whether it is valid or modified. Because a cache cannot hold all memory, an incoming line may evict an older one chosen by a replacement policy. If the evicted line contains changes not yet written to lower memory, those bytes must be written back. Associativity gives an address several possible locations within a set, reducing collisions compared with a single fixed slot but increasing lookup complexity.
Multiple cores introduce coherence. If one core modifies a line that another core has cached, they must not continue using inconsistent values. A coherence protocol exchanges ownership and invalidation messages so cores agree on the visible state. False sharing occurs when independent variables used by different threads occupy the same line; frequent writes then move or invalidate the whole line even though the threads do not logically share data. Layout and synchronization can therefore matter as much as cache size.
Processors also prefetch lines they predict will be needed soon. Streaming patterns are easy to detect, while irregular pointer chasing is difficult. A useful prefetch overlaps memory delay with current work. An inaccurate one wastes bandwidth, displaces valuable data, and consumes power. Software can improve locality through compact data structures, contiguous traversal, and blocking a large problem into chunks that fit a cache. Performance counters reveal misses, replacements, and stalled cycles.
The cache speeds a program by turning repeated or nearby memory requests into short local transactions. Its success depends on the working set fitting long enough to be reused and on access patterns that hardware can recognize. A larger cache is not automatically faster because longer lookup time and power can offset capacity. The hierarchy, line size, replacement rules, prefetchers, and coherence system work together to present one shared memory while serving most requests from much closer silicon.
Explore more "Explainers"
Discover additional explainers across politics, science, business, technology, and other fields. Each explainer breaks down a complex idea into clear, everyday language—helping you better understand how major concepts, systems, and debates shape the world around us.
