# What is AI inference? Definition and business implications

> Inference is the usage phase of an AI model, during which the model computes a response from a given prompt. It is the operation billed by API providers, distinct from training which is an initial fixed cost.

## Definition
Inference consists of passing a query through the model's neural network to generate an output. For an LLM, this amounts to predicting the response tokens one by one, each token requiring a full pass through the network. The larger the model (in parameter count), the more calculations each pass demands, and the higher the latency. Three variables condition inference cost: the size of the model (more parameters means more calculations per token), the number of input and output tokens (direct proportion), and the numerical precision used (FP32, FP16, INT8, INT4). Quantisation reduces precision without significantly degrading quality, and proportionally divides memory consumption and compute cost. Inference is the recurring cost of an AI application: it is paid on every request, whereas training is an upfront cost.

## Example
According to the Stanford AI Index 2025 report, the inference cost for a GPT-3.5-level model (MMLU score 64.8) fell from 20 dollars per million tokens in November 2022 to 0.07 dollar per million in October 2024, a reduction of more than 280-fold in under two years. The same level of quality is today available at a much lower cost. This continuous fall changes the economic arbitration: what was out of reach in 2023 is now a secondary cost item.

## For a leader
Inference is the only recurring AI budget line, and it grows as industrialised usage increases. Three implications for the executive. First, the spectacular fall in costs (by more than 280 in under two years for GPT-3.5 quality) changes the economic arbitration: use cases that were not viable yesterday now become so. Auditing the cases discarded in 2022-2023 for poor ROI may reveal new opportunities. Second, inference is sensitive to prompting discipline: shorter prompts, calibrated contexts, and use of smaller models when complexity does not warrant a flagship. Usage sobriety is a direct economic lever. Third, local inference on internal infrastructure becomes feasible for data-sensitive cases: a quantised Mistral 7B runs on standard hardware, at very low marginal cost after the initial investment.

## FAQ

**What is inference, and how does it differ from training?**
Inference is the usage phase of the model: it computes a response from a given prompt. It is the operation API providers bill, on every request. Training, by contrast, is an upfront fixed cost, incurred only once.

**What drives inference cost up?**
Three variables: the model's size (more parameters means more calculations per token), the number of input and output tokens (a direct proportion), and the numerical precision used (FP32, FP16, INT8, INT4). Quantisation lowers that precision without significantly degrading quality, and proportionally divides memory and compute cost.

**Why is inference the budget line to watch?**
Because it is the only recurring AI cost, and it grows as usage industrialises. Training is paid once; inference is paid on every request.

**Does the fall in inference costs change anything for us?**
A great deal. According to the Stanford AI Index 2025, the cost for GPT-3.5-equivalent quality fell from 20 dollars per million tokens in November 2022 to 0.07 dollar in October 2024, a reduction of more than 280-fold in under two years. Use cases discarded in 2022-2023 for poor ROI are worth re-auditing: what was once out of reach has become a secondary cost item.

**Can we run inference in-house for sensitive data?**
Yes, it is now feasible. A quantised lightweight model, such as Mistral 7B, runs on standard hardware at very low marginal cost once the initial investment is made. It is a sound option where data confidentiality comes first.

## See also

- [Training](https://wyp.agency/en/glossary/training/)
- [Token](https://wyp.agency/en/glossary/token/)
- [Parameter](https://wyp.agency/en/glossary/parameter/)
- [LLM (Large Language Model)](https://wyp.agency/en/glossary/llm/)
- [Foundation model](https://wyp.agency/en/glossary/foundation-model/)
- [Reasoning model](https://wyp.agency/en/glossary/reasoning-model/)

## Sources

- Artificial Intelligence Index Report 2025, Stanford HAI, chapter 1 (https://hai.stanford.edu/ai-index/2025-ai-index-report)
- Claude API public pricing (Anthropic) (https://claude.com/pricing)
- Artificial Intelligence Index Report 2026, Stanford HAI, chapter 1 (Research and Development) (https://hai.stanford.edu/ai-index/2026-ai-index-report/research-and-development)

---
Entry from the WYP.agency AI glossary · https://wyp.agency/en/glossary/inference/ · Last updated : 2026-10-14