Neural Tangent Kernel (NTK)

Neural networks are notoriously difficult to analyse mathematically. They are non-linear, high-dimensional, and their training dynamics are complex. For decades, theoretical understanding lagged far behind practical results: networks worked much better than theory suggested they should, and practitioners had more intuition than rigorous explanation. The Neural Tangent Kernel, introduced in 2018, provided a new theoretical lens.

The key insight is what happens to a neural network in the limit of infinite width - as the number of neurons in each hidden layer grows without bound. In this limit, something surprising occurs: the network''s behaviour during gradient descent training becomes linear in a particular way, and the learning dynamics can be completely described by a fixed kernel function called the Neural Tangent Kernel. The NTK captures how the network''s output changes in response to a small parameter update at any given point in training.

In the infinite-width limit, the NTK remains constant throughout training - it does not change as the weights update. This means the network learns in a kernel regression regime: it effectively fits a function in a reproducing kernel Hilbert space defined by the NTK. This is a much more tractable mathematical object than a general neural network, and many properties of training can be derived analytically.

The NTK framework explains several puzzling empirical observations. Why do overparameterised networks (networks with far more parameters than training examples) generalise well rather than overfitting? In the NTK regime, overparameterisation is actually beneficial - it enables interpolation of training data while maintaining a bias toward smooth, structured functions. Why does random initialisation produce networks that train reliably? The NTK at initialisation describes what the network can learn, and random initialisation in the right regime produces NTKs with good coverage of function space.

The practical significance of NTK theory is that it provides intuitions that transfer to finite-width real networks. Understanding why infinite networks behave in certain ways illuminates why finite networks behave similarly, and NTK-inspired analysis has guided practical decisions about initialisation, learning rate scaling, and architecture design.',

The framework has limitations: real networks operate far from the infinite-width limit, and the NTK changes during training of finite networks. But as a first-principles theory of learning, NTK provided the field with tools for building on top of, and motivated research into when and why the lazy training regime (where NTK approximation holds) is or is not a good model of real network training.