Deep Learning

The word "deep" in deep learning refers to depth: neural networks with many layers stacked on top of each other. A shallow network might have one or two hidden layers between the input and output. A deep network might have dozens, hundreds, or in the largest modern models, effectively thousands of layers of computation.

Why does depth matter? Depth enables hierarchical abstraction. Each layer in a deep network can learn to detect features that earlier layers missed, building increasingly complex representations from simpler ones. In vision, shallow networks see pixels; deep networks see objects. In language, shallow networks see words; deep networks see meaning, context, and intent.

The possibility of training deep networks had been understood for decades, but it was practically impossible. Networks with many layers suffered from a problem called vanishing gradients: during training, the error signal that flows back through the network to update weights became weaker with each layer it passed through, leaving early layers nearly unable to learn.

Several breakthroughs unlocked deep learning in the early 2010s. Better activation functions (ReLU replaced sigmoid), better weight initialisation, and dramatic increases in compute and data were all important. So was the GPU: graphics processing units turned out to be extraordinarily efficient at the parallel matrix operations that neural networks require, accelerating training by orders of magnitude.

The defining moment came in 2012, when AlexNet - a deep convolutional neural network - won the ImageNet competition by a margin that shocked researchers. Computer vision transformed almost immediately. Text followed with the introduction of transformers in 2017. By the early 2020s, deep learning dominated essentially every area of machine learning research and application.