Mixed Precision Training

Neural network training and inference involve enormous numbers of floating-point operations. The standard 32-bit floating-point format (FP32) uses 4 bytes per value. Training a 7-billion-parameter model with FP32 requires 28GB just for the weights - before accounting for gradients, optimiser states, and activations. Mixed precision training is the now-standard approach of using lower-precision formats where possible to reduce memory footprint and increase computational throughput.

Modern GPUs have specialised hardware (Tensor Cores on NVIDIA GPUs, Matrix Engines on other accelerators) that can execute 16-bit floating-point matrix operations far faster than 32-bit operations - up to 8x faster for certain operations on the H100. Mixed precision exploits this by using 16-bit formats for the bulk of computation while retaining 32-bit precision where it matters for training stability.

The standard mixed precision approach: maintain FP32 master copies of model weights (the authoritative parameters). For each forward and backward pass, cast the weights to FP16 or BF16 for computation. Accumulate gradients in FP32. Update the FP32 master weights. The FP16/BF16 copies used for computation are ephemeral; only the FP32 masters are permanently stored. This sounds like it wastes memory on duplicate weights, but the computation (which dominates training time) runs at 16-bit speed, and activations and gradients - which are the majority of memory consumption during training - are stored in 16-bit.

FP16 and BF16 are distinct 16-bit formats with different tradeoffs. FP16 has more mantissa bits (more precision in the represented value) but a narrower exponent range, making it prone to overflow (values too large for the format) and underflow (values too small, rounded to zero) during gradient computations. BF16 has fewer mantissa bits but the same exponent range as FP32, making it more numerically stable for training. Modern hardware almost universally uses BF16 for training and FP16 is now primarily used for inference.

FP16 training requires loss scaling to address the underflow problem: gradients are multiplied by a large scale factor before the backward pass to shift them into the representable range, then divided back out before the weight update. This prevents small gradient values from vanishing to zero in FP16 representation. Dynamic loss scaling automatically adjusts the scale factor based on whether gradient overflow is detected.

Beyond training, quantised inference uses even lower precision (INT8, INT4, or 4-bit NF4 formats) for serving deployed models, trading a small amount of model quality for dramatic memory and throughput improvements.