Batches and Epochs Explainer
A batch is the group of training examples used to calculate one parameter update. An epoch is one complete pass through the available training dataset. If a dataset has 10,000 examples and the batch size is 100, one epoch contains about 100 update steps.
Smaller batches require less memory and produce noisier gradient estimates. Larger batches can use parallel hardware efficiently and give smoother estimates, but they consume more memory and may require changes to the learning rate. The best choice depends on the model, data, hardware, and optimization method.
Training for multiple epochs lets a model revisit examples as its parameters change. Data are commonly shuffled between passes so batches do not repeat in the same order. With streaming or extremely large datasets, teams may measure progress in steps or tokens rather than meaningful full epochs.
More epochs are not always better. Training loss can keep falling while validation performance stops improving or worsens as the model overfits. Early stopping, checkpoint comparisons, and held-out evaluation help determine when additional training is no longer useful.
A batch is a subset used for one training update, while an epoch is one pass through the training dataset.
Key terms include batch, mini-batch, batch size, epoch, training step, shuffle, gradient estimate, checkpoint and early stopping.
Each batch produces a loss and gradient estimate, after which the optimizer updates the model before processing the next batch.
The final smaller batch may contain fewer examples when the dataset size is not evenly divisible by the selected batch size.
Epochs are intuitive for fixed datasets, but steps and processed tokens can be more comparable when datasets are sampled, filtered, or continuously refreshed.
The useful stopping point is based on validated performance and stability, not on a universal number of epochs.
Explore more "Explainers"
Discover additional explainers across politics, science, business, technology, and other fields. Each explainer breaks down a complex idea into clear, everyday language—helping you better understand how major concepts, systems, and debates shape the world around us.