Model Pruning

Large neural networks are notoriously over-parameterised: they have far more parameters than the minimum needed to represent the function they have learned. Pruning exploits this by identifying parameters that are near-zero or contribute minimally to model outputs and removing them, producing a smaller, faster model that ideally retains most of the original model's quality.

The motivation is practical. A model with 70% of its weights removed runs faster, uses less memory, and can be deployed on hardware that could not fit the original. For edge deployment - running models on phones, embedded devices, or IoT hardware - pruning is often essential rather than optional.

Pruning strategies vary along two key axes: what is pruned and when. Magnitude pruning (the simplest approach) removes weights whose absolute values fall below a threshold - the intuition being that small weights contribute little to model outputs. More sophisticated importance-based pruning estimates each weight's contribution by measuring how much removing it increases the loss, using gradient information or second-order approximations like OBS (Optimal Brain Surgeon) or OBD (Optimal Brain Damage).

Structured versus unstructured pruning reflects the hardware implications. Unstructured (weight-level) pruning removes individual weights, creating sparse weight matrices. While this achieves high compression ratios, sparse matrix operations are not well-supported by standard GPU hardware, which is optimised for dense operations. Realising speedups from unstructured pruning requires specialised sparse execution libraries or hardware. Structured pruning removes entire components - neurons, attention heads, layers, or convolutional filters - producing dense (non-sparse) weight matrices that execute efficiently on standard hardware without specialised support. The tradeoff: structured pruning achieves lower compression ratios for the same quality loss.

Pruning timing also varies. Post-training pruning applies pruning to an already-trained model, often followed by a fine-tuning phase to recover the quality lost from pruning. Iterative pruning alternates between pruning steps and fine-tuning, gradually increasing sparsity. Pruning during training (sparse training) enforces sparsity constraints throughout the training process.

The Lottery Ticket Hypothesis (Frankle & Carlin, 2018) proposed that large networks contain small subnetworks ("winning tickets") that can be trained from scratch to match the full network's performance. While this is theoretically compelling, finding these tickets reliably at scale has proven difficult.

For large language models, attention head pruning removes entire attention heads that contribute minimally to outputs, and layer dropping removes full transformer layers. Both have been demonstrated to reduce model size by 20-40% with small quality losses in distillation pipelines.