Last reviewed:
What is an AI parameter? Definition and business implications
A parameter of an AI model is a numerical value learned during training that determines the strength of connections between the artificial neurons of the network. The parameter count measures the model's capacity to store associations between the elements of its training corpus.
An artificial neural network is made up of layers of neurons linked by connections, each weighted by a numerical value. These weights are the model's parameters. At the start of training, they are initialised randomly. With each example processed, the backpropagation algorithm slightly adjusts these weights to reduce the error between the model's prediction and the expected answer. After billions of iterations, the parameters converge towards values that encode the statistical regularities of the corpus. The higher the parameter count, the more fine-grained associations the model can store, but the more its training costs (in GPU, electricity, data) and the slower its inference. A technique called quantisation makes it possible to reduce the numerical precision of each parameter (from 32 to 8 or 4 bits), dividing the required memory by 4 to 8, generally without significant degradation in quality.
Concrete example
In 2017, the original transformer had 65 million parameters and fit on a 16 GB GPU. Today, a Llama 3 model with 70 billion parameters requires 280 GB of memory at full precision (FP32), or 35 GB at 4-bit quantisation (INT4). This compression makes it possible to deploy near-equivalent quality models on standard servers: a quantised Mistral 7B runs on a MacBook Pro M3 with less than 4 GB of RAM, whereas GPT-3 (175 billion parameters) requires a specialised cluster worth several hundred thousand euros.
See also
Sources
- Language Models are Few-Shot Learners, Brown et al., NeurIPS 2020. https://arxiv.org/abs/2005.14165
- The Llama 3 Herd of Models, Meta AI, 2024. https://ai.meta.com/research/publications/the-llama-3-herd-of-models/