Transfer Learning

Transfer learning is the core principle behind modern AI development: rather than training a model from random initialisation for every new task, you start from a model that has already learned useful representations on a related task, then adapt it to your specific need. The transferred knowledge provides a starting point that dramatically reduces the data, time, and compute required to reach good performance.

The insight that makes transfer learning work is that many tasks share underlying structure. A model trained to predict the next word in English text learns representations of grammar, semantics, world knowledge, and reasoning that are useful for almost any language task. A model trained to recognise objects in photographs learns low-level features like edges and textures that are useful for almost any vision task. The early layers of a well-trained model capture general features; later layers capture task-specific features. Transfer learning reuses the general features and replaces or adapts only the task-specific ones.

In the pre-transformer era, transfer learning in NLP was limited. Models were relatively shallow, and training from scratch on task-specific labelled data was often feasible. The paradigm shift came with large pre-trained models: when BERT showed that a model pre-trained on massive text corpora could be fine-tuned to state-of-the-art performance on almost any NLP task with just a small labelled dataset, transfer learning became the dominant paradigm. GPT took this further, showing that pre-training at sufficient scale produced models useful for few-shot and even zero-shot tasks - no task-specific fine-tuning required.

Transfer learning has a natural hierarchy: the more similar the source and target tasks, the more complete the transfer. A model trained on general English text transfers well to sentiment analysis. A model trained on medical literature transfers better still to clinical note analysis. A model trained specifically on radiology reports transfers best of all to chest X-ray interpretation. Each level of domain specificity reduces the data needed for adaptation while increasing performance on the target task.

The limits of transfer learning are also instructive. Capabilities that were not present in pre-training data do not transfer - a model that never saw code cannot code, regardless of how extensively you fine-tune it on a different task. Transfer learning moves capabilities around; it does not create them from nothing.