Backpropagation Explainer
Backpropagation is the procedure used to compute gradients through a neural network. After a forward pass produces an output and a loss, backpropagation works backward through the recorded operations to determine how a small change in each parameter would affect that loss.
The method repeatedly applies the calculus chain rule. Each layer receives a gradient from later operations, combines it with the layer’s local derivatives, and passes the result toward earlier layers. Shared intermediate calculations make this far more efficient than perturbing millions of parameters one at a time.
Backpropagation calculates gradients; it does not decide the update by itself. An optimizer such as stochastic gradient descent or Adam uses those gradients to change the parameters. Training repeats forward pass, loss calculation, backward pass, and update across many batches.
Deep networks can have vanishing or exploding gradients, numerical errors, and operations that are difficult to differentiate. Activation choices, normalization, residual connections, initialization, and gradient clipping can help. A correct backward pass is still only as useful as the data, objective, and evaluation surrounding it.
Backpropagation efficiently calculates how each model parameter contributed to the training loss.
Key terms include forward pass, backward pass, chain rule, derivative, computational graph, gradient, parameter, optimizer and automatic differentiation.
Backpropagation assigns mathematical credit or blame by tracing the loss through every differentiable operation that produced it.
Automatic-differentiation systems store the computation graph during the forward pass and reuse it to calculate gradients in reverse order.
The optimizer determines how gradients become parameter updates, while the loss function determines what behavior receives a favorable signal.
Gradient checks and unit tests can catch implementation errors, but they cannot reveal whether the chosen objective matches the real-world goal.
Explore more "Explainers"
Discover additional explainers across politics, science, business, technology, and other fields. Each explainer breaks down a complex idea into clear, everyday language—helping you better understand how major concepts, systems, and debates shape the world around us.