Last reviewed:
What is the transformer architecture? Definition and business implications
The transformer is the neural-network architecture, introduced by Google in 2017, that underpins nearly all current generative AI models. Its central innovation is the attention mechanism, which lets the model dynamically weigh the relative importance of words in a sequence.
Before 2017, language-processing models used sequential architectures (RNNs, LSTMs) that processed words one by one, in order. This approach limited the model's ability to capture relationships between distant words in a text, and made training hard to parallelise. The transformer architecture, presented in the paper Attention Is All You Need (Vaswani et al., 2017), introduces a break: the model processes all words simultaneously, computing for each pair of words an attention score that measures their mutual relevance. Consequence: parallelisation becomes possible (training shifts from sequential to massively parallel, GPU-exploitable), and the model can establish very-long-distance links in a text. The transformer is today the foundation of all major LLMs (GPT, Claude, Gemini, Llama, Mistral), as well as of image-generation models (DALL-E, Stable Diffusion) and multimodal models.
Concrete example
The original transformer paper was published by eight Google researchers in June 2017. It described a 65-million-parameter model trained for English-German translation. Nearly nine years later, in 2026, the transformer architecture remains the dominant architecture for all foundation models published by the major laboratories (OpenAI, Anthropic, Google DeepMind, Meta AI, Mistral). Architectural variants (encoder-only like BERT, decoder-only like GPT, mixture of experts like Mixtral) are all evolutions of the original transformer. No competing architecture (Mamba, RWKV) has achieved comparable industrial adoption, despite regular technical promises.
See also
Further reading
Attention Is All You Need, original paper (Vaswani et al., 2017)
Sources
- Attention Is All You Need, Vaswani et al., NeurIPS 2017. https://arxiv.org/abs/1706.03762