Mixed-Precision Training Explainer
Mixed-precision training performs much of neural-network training with lower-precision numbers while keeping selected operations or state in higher precision. Common implementations combine 16-bit formats with 32-bit master weights or accumulations.
Lower precision reduces memory traffic and can use specialized accelerator hardware for faster matrix operations. Automatic casting chooses suitable formats for different operations, while loss scaling helps prevent small gradients from underflowing in some formats.
The technique can increase throughput and allow larger models or batches without materially changing final accuracy when configured well. Modern frameworks automate many casting and scaling decisions.
Numerical range still matters. Overflow, underflow, unstable reductions, and unsupported operations can produce divergence or silent errors. Teams should monitor finite losses and gradients, compare with a higher-precision baseline, and understand differences between numeric formats.
Mixed-precision training uses lower-precision arithmetic where it is safe and higher precision where numerical range or stability is needed.
Key terms include mixed precision, floating point, FP16, BF16, FP32, automatic casting, loss scaling, underflow and master weights.
Fast lower-precision arithmetic handles many matrix operations and reduces memory use.
Higher precision protects sensitive accumulations, updates, or values with a wider numeric range.
Overflow or underflow can corrupt training even when the run appears to continue.
Finite-value checks and comparisons with a trusted baseline help reveal precision failures.
Explore more "Explainers"
Discover additional explainers across politics, science, business, technology, and other fields. Each explainer breaks down a complex idea into clear, everyday language—helping you better understand how major concepts, systems, and debates shape the world around us.