# What is the transformer architecture? Definition and business implications

> The transformer is the neural-network architecture, introduced by Google in 2017, that underpins nearly all current generative AI models. Its central innovation is the attention mechanism, which lets the model dynamically weigh the relative importance of words in a sequence.

## Definition
Before 2017, language-processing models used sequential architectures (RNNs, LSTMs) that processed words one by one, in order. This approach limited the model's ability to capture relationships between distant words in a text, and made training hard to parallelise. The transformer architecture, presented in the paper *Attention Is All You Need* (Vaswani et al., 2017), introduces a break: the model processes all words simultaneously, computing for each pair of words an attention score that measures their mutual relevance. Consequence: parallelisation becomes possible (training shifts from sequential to massively parallel, GPU-exploitable), and the model can establish very-long-distance links in a text. The transformer is today the foundation of all major LLMs (GPT, Claude, Gemini, Llama, Mistral), as well as of image-generation models (DALL-E, Stable Diffusion) and multimodal models.

## Example
The original transformer paper was published by eight Google researchers in June 2017. It described a 65-million-parameter model trained for English-German translation. More than nine years later, in 2026, the transformer architecture remains the dominant architecture for all foundation models published by the major laboratories (OpenAI, Anthropic, Google DeepMind, Meta AI, Mistral). Architectural variants (encoder-only like BERT, decoder-only like GPT, mixture of experts like Mixtral) are all evolutions of the original transformer. Competing architectures (Mamba, RWKV) have not displaced the transformer. They mostly appear in hybrid models: in 2025 NVIDIA published Nemotron-H, which replaces most attention layers with Mamba layers and reports up to 3 times faster inference.

## For a leader
“Is the model you are proposing actually based on the transformer architecture?” The question may seem trivial; it is not. Three sub-questions to chain. First, is it an encoder-only transformer (useful for semantic search), decoder-only (standard LLM), or hybrid? The architecture type conditions the nature of addressable use cases. Second, does the model use recent optimisations (Flash Attention, grouped-query attention) that reduce inference memory consumption? Grouped-query attention shrinks the inference memory cache, and Flash Attention speeds up the attention computation. The gain in serving cost depends on the model. Third, is your provider able to explain the architectural trade-offs made on your deployment? If the answer is vague, that is a signal about their actual mastery of the underlying technical stack.

## FAQ

**What is the transformer architecture?**
It is the neural-network architecture, introduced by Google in 2017, that underpins nearly all current generative AI models. Its central innovation is the attention mechanism, which lets the model dynamically weigh the relative importance of words in a sequence.

**What did it change compared with earlier architectures?**
Before 2017, models (RNNs, LSTMs) processed words one by one, in order, which limited long-distance relationships and made training hard to parallelise. The transformer processes all words at once, computing an attention score for each pair of words. The result: training becomes massively parallel and GPU-exploitable, and the model can forge very-long-distance links in a text.

**Is it used only for text?**
No. The transformer is the basis of all major LLMs (GPT, Claude, Gemini, Llama, Mistral), as well as of image-generation models (DALL-E, Stable Diffusion) and multimodal models. In 2026, more than nine years after the original paper, it remains the dominant architecture for every foundation model from the major labs.

**Are all variants equivalent for our use cases?**
No, and it is a question to put to your provider. An encoder-only transformer (BERT-type) mainly serves semantic search, a decoder-only one (GPT-type) is a standard LLM, and there are mixture-of-experts variants (Mixtral-type). The architecture type conditions the nature of addressable use cases.

**How do you tell whether a model is well optimised?**
Check whether it uses recent optimisations such as Flash Attention or grouped-query attention, which cut memory consumption and speed up the attention computation, with a serving-cost gain that depends on the model. If your provider cannot spell out the architectural trade-offs made on your deployment, that is a signal about their real command of the underlying stack.

## See also

- [LLM (Large Language Model)](https://wyp.agency/en/glossary/llm/)
- [Parameter](https://wyp.agency/en/glossary/parameter/)
- [Training](https://wyp.agency/en/glossary/training/)
- [Foundation model](https://wyp.agency/en/glossary/foundation-model/)
- [Generative AI](https://wyp.agency/en/glossary/generative-ai/)
- [NLP (natural language processing)](https://wyp.agency/en/glossary/natural-language-processing/)
- [Deep learning](https://wyp.agency/en/glossary/deep-learning/)
- [Neural network (artificial)](https://wyp.agency/en/glossary/neural-network/)

## Sources

- Attention Is All You Need, Vaswani et al., NeurIPS 2017 (https://arxiv.org/abs/1706.03762)
- Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models, NVIDIA, 2025 (https://arxiv.org/abs/2504.03624)
- FlashAttention, Dao et al., 2022 (https://arxiv.org/abs/2205.14135)
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, Ainslie et al., 2023 (https://arxiv.org/abs/2305.13245)

---
Entry from the WYP.agency AI glossary · https://wyp.agency/en/glossary/transformer/ · Last updated : 2026-10-14