Parameters

A neural network is, at its mathematical core, a function with millions or billions of tunable numbers - its parameters, also called weights. During training, these numbers are adjusted repeatedly to make the model's predictions more accurate. Once training is complete, the parameters are frozen, encoding everything the model has learned into a massive array of floating-point numbers.

Parameters exist at every layer of a neural network. In a standard layer, each connection between nodes has its own weight - a number that determines how much that connection's input influences the next layer. There are also bias terms, additional parameters added to each node's output before activation. The total count of all weights and biases is the model's parameter count.

This count has become a common shorthand for model scale: "a 7 billion parameter model" or "a 70 billion parameter model." Larger parameter counts generally allow more complex patterns to be learned, but require more data, more compute, and more memory to train and run. GPT-3 had 175 billion parameters; estimates for GPT-4 are in the hundreds of billions to over a trillion.

In transformers - the architecture underlying modern large language models - the key parameters are the matrices in the attention and feedforward layers. Multiple large matrices per layer store the model's learned knowledge. It is in these matrices that the model's understanding of language, facts, and reasoning is distributed.

An important and strange fact: no individual parameter encodes a specific fact. The information in a language model is distributed across billions of parameters, with no single parameter representing "Paris is the capital of France." The knowledge is a property of the collective. This distributed representation is what makes neural networks robust - and what makes understanding exactly what they know so difficult.