What is AI inference? Definition and business implications

Inference consists of passing a query through the model's neural network to generate an output. For an LLM, this amounts to predicting the response tokens one by one, each token requiring a full pass through the network. The larger the model (in parameter count), the more calculations each pass demands, and the higher the latency. Three variables condition inference cost: the size of the model (more parameters means more calculations per token), the number of input and output tokens (direct proportion), and the numerical precision used (FP32, FP16, INT8, INT4). Quantisation reduces precision without significantly degrading quality, and proportionally divides memory consumption and compute cost. Inference accounts for the bulk of an AI application's production cost: according to NVIDIA's public analyses, around 80% of an enterprise's operational AI budget at scale goes to inference, versus 20% to training.