Last reviewed:
What is AI inference? Definition and business implications
Inference is the usage phase of an AI model, during which the model computes a response from a given prompt. It is the operation billed by API providers, distinct from training which is an initial fixed cost.
Inference consists of passing a query through the model's neural network to generate an output. For an LLM, this amounts to predicting the response tokens one by one, each token requiring a full pass through the network. The larger the model (in parameter count), the more calculations each pass demands, and the higher the latency. Three variables condition inference cost: the size of the model (more parameters means more calculations per token), the number of input and output tokens (direct proportion), and the numerical precision used (FP32, FP16, INT8, INT4). Quantisation reduces precision without significantly degrading quality, and proportionally divides memory consumption and compute cost. Inference is the recurring cost of an AI application: it is paid on every request, whereas training is an upfront cost.
Concrete example
According to the Stanford AI Index 2025 report, the inference cost for a GPT-3.5-level model (MMLU score 64.8) fell from 20 dollars per million tokens in November 2022 to 0.07 dollar per million in October 2024, a reduction of more than 280-fold in under two years. The same level of quality is today available at a much lower cost. This continuous fall changes the economic arbitration: what was out of reach in 2023 is now a secondary cost item.
FAQ
What is inference, and how does it differ from training?
Inference is the usage phase of the model: it computes a response from a given prompt. It is the operation API providers bill, on every request. Training, by contrast, is an upfront fixed cost, incurred only once.
What drives inference cost up?
Three variables: the model's size (more parameters means more calculations per token), the number of input and output tokens (a direct proportion), and the numerical precision used (FP32, FP16, INT8, INT4). Quantisation lowers that precision without significantly degrading quality, and proportionally divides memory and compute cost.
Why is inference the budget line to watch?
Because it is the only recurring AI cost, and it grows as usage industrialises. Training is paid once; inference is paid on every request.
Does the fall in inference costs change anything for us?
A great deal. According to the Stanford AI Index 2025, the cost for GPT-3.5-equivalent quality fell from 20 dollars per million tokens in November 2022 to 0.07 dollar in October 2024, a reduction of more than 280-fold in under two years. Use cases discarded in 2022-2023 for poor ROI are worth re-auditing: what was once out of reach has become a secondary cost item.
Can we run inference in-house for sensitive data?
Yes, it is now feasible. A quantised lightweight model, such as Mistral 7B, runs on standard hardware at very low marginal cost once the initial investment is made. It is a sound option where data confidentiality comes first.
See also
Sources
- Artificial Intelligence Index Report 2025, Stanford HAI, chapter 1. https://hai.stanford.edu/ai-index/2025-ai-index-report
- Claude API public pricing (Anthropic). https://claude.com/pricing
- Artificial Intelligence Index Report 2026, Stanford HAI, chapter 1 (Research and Development). https://hai.stanford.edu/ai-index/2026-ai-index-report/research-and-development