What Is Gradient Clipping?

Gradient Clipping Explainer

Gradient clipping limits the magnitude of gradients before an optimizer updates model parameters. It is commonly used when gradients occasionally become extremely large, a problem that can destabilize training or produce numerical overflow.

Norm clipping rescales the whole gradient vector when its norm exceeds a threshold, preserving its direction while shrinking its length. Value clipping instead caps individual components. The two methods change optimization differently and should not be treated as interchangeable.

Clipping became especially familiar in recurrent networks, where repeated transformations can produce exploding gradients. It is also used in large neural models and privacy-preserving training, although the purpose and threshold may differ across those settings.

A threshold that is too low can suppress useful learning, while one that is too high may rarely activate. Teams monitor gradient norms and clipping frequency, then investigate root causes such as an excessive learning rate, poor scaling, unstable loss, or faulty data.

Gradient clipping limits the size of gradients or parameter updates to reduce instability caused by exploding gradients.

Key terms include gradient norm, threshold, exploding gradient, rescaling, value clipping, optimizer, numerical stability, update and monitoring.

Norm clipping rescales a gradient vector only when the chosen norm exceeds a threshold, while value clipping limits each component separately.

The selected method changes both the magnitude and sometimes the direction of the update.

Clipping can prevent a rare extreme update from ruining a run, but it does not explain why the gradient became extreme.

Logs should record gradient norms and clipping events so developers can distinguish a healthy safeguard from a persistent training problem.

Explore more "Explainers"

Discover additional explainers across politics, science, business, technology, and other fields. Each explainer breaks down a complex idea into clear, everyday language—helping you better understand how major concepts, systems, and debates shape the world around us.