Gradient Accumulation Explainer
Gradient accumulation is a training technique that processes several small microbatches before applying one optimizer update. Gradients from each backward pass are added together rather than clearing them after every microbatch.
After the chosen number of accumulation steps, the optimizer updates the weights and the stored gradients are reset. The loss is commonly scaled so the accumulated result corresponds to the intended effective batch size.
This lets a model train with an effective batch larger than device memory could hold at once. It can make experiments feasible on smaller accelerators and separate memory constraints from the desired optimization batch size.
Accumulation does not reproduce every behavior of a physically larger batch. Batch-dependent layers, numerical precision, distributed synchronization, learning-rate schedules, and optimizer step counts can differ. Logging should distinguish microsteps from actual updates.
Gradient accumulation adds gradients from several smaller microbatches before updating model weights, approximating a larger batch with less memory.
Key terms include gradient, accumulation step, microbatch, effective batch size, backward pass, optimizer update, loss scaling and synchronization.
Each microbatch contributes gradients while the model weights stay unchanged.
The optimizer steps only after the configured accumulation count is reached.
Batch statistics, floating-point rounding, schedules, and distributed communication can differ.
Metrics should count optimizer updates separately from processed microbatches.
Explore more "Explainers"
Discover additional explainers across politics, science, business, technology, and other fields. Each explainer breaks down a complex idea into clear, everyday language—helping you better understand how major concepts, systems, and debates shape the world around us.
