Activation Checkpointing Explainer
Activation checkpointing is a memory-saving method for neural-network training. A normal forward pass stores many intermediate activations because the backward pass needs them to calculate gradients.
With checkpointing, training retains only selected activations. When backpropagation reaches a missing segment, the system reruns that part of the forward computation to reconstruct the required values and then continues computing gradients.
The method reduces peak memory and can make deeper models, longer sequences, or larger batches fit on available hardware. Checkpoint placement controls the balance between saved memory and repeated computation.
Recomputation increases runtime and can interact with randomness, stateful layers, offloading, and distributed schedules. Implementations must reproduce the same forward behavior, preserve random-number state when needed, and benchmark end-to-end throughput rather than memory alone.
Activation checkpointing saves only selected intermediate states during training and recomputes the others during backpropagation to reduce memory use.
Key terms include activation, checkpoint, forward pass, backward pass, gradient, recomputation, peak memory, segment and random state.
Only selected activations are stored during the original forward pass.
Missing activations are recomputed later when the backward pass requires them.
Random or stateful operations can break correctness if the repeated forward segment differs.
Checkpoint placement should be evaluated for both peak memory and total training speed.
Explore more "Explainers"
Discover additional explainers across politics, science, business, technology, and other fields. Each explainer breaks down a complex idea into clear, everyday language—helping you better understand how major concepts, systems, and debates shape the world around us.