Last reviewed:
What is MoE (Mixture of Experts)? Definition and business implications
Mixture of Experts (MoE) is an AI model architecture that splits the network into specialised sub-models, called experts. For each token processed, a router dynamically selects a few experts, leaving the others inactive. The model has the capacity of a large model but the compute cost of a smaller one.
The idea of mixture of experts is old; it was applied to transformers at very large scale by Fedus, Zoph, and Shazeer in 2021 (Switch Transformer, Google). The idea: instead of activating the entire network on each inference, a lightweight router directs each token to a subset of specialised experts, typically 1 to 8 out of the dozens available. Consequence: one can train and deploy models with hundreds of billions of parameters, but only a small fraction is mobilised on each token (37 billion out of 671 for DeepSeek V3). In 2026, most large open-weight models rely on an MoE architecture (DeepSeek V4, Llama 4, Qwen 3.5, Mistral Large 3); providers of closed models generally do not publish their architecture. DeepSeek V3: 671 billion total parameters, 37 billion active per token. Mixtral 8x22B (Mistral): 141 billion total, 39 billion active. The competitive advantage of MoE is so clear that it has become an architectural standard for very-large-scale models.
Concrete example
DeepSeek V3 illustrates the economic advantage of MoE. The model has 671 billion parameters in total, but activates only 37 billion per token, that is, 5.5% of the whole. According to DeepSeek, it achieves performance comparable to the best closed models of the time, with a full training run of 2.788 million H800 GPU hours (DeepSeek technical report). This efficiency explains the massive adoption of MoE by large open-weight models in 2025-2026.
FAQ
What is an MoE (Mixture of Experts) model?
It is an architecture that splits the network into specialised sub-models called experts. For each token processed, a router dynamically selects a few experts, typically 1 to 8 out of the dozens available, and leaves the others inactive. The model thus has the capacity of a large model but the compute cost of a smaller one.
Why is an MoE model cheaper to run?
Because only a fraction of the parameters is activated per token. DeepSeek V3, for example, has 671 billion parameters in total but activates only 37 billion per token, that is 5.5%.
Why does the dense-versus-MoE distinction matter for comparing prices?
Because the inference cost per token may reflect actual activation (for example 5% of parameters) or the model's total size. Without that distinction, a price comparison between models is misleading. It is one of the first questions to ask your IT director or provider.
What risks does an MoE model carry?
Two watch points. First, routing collapse, a well-documented effect where traffic concentrates on a few experts and degrades average quality, which is why a load-stability test matters. Second, infrastructure: an MoE model with 671 billion parameters requires several hundred gigabytes of VRAM, even if activation stays minimal.
Is MoE widespread today?
Yes. In 2026, most large open-weight models rely on an MoE architecture, such as DeepSeek V4, Llama 4, Qwen 3.5 and Mistral Large 3. The competitive advantage is so clear that MoE has become an architectural standard for very-large-scale models.
See also
Further reading
Switch Transformers: Scaling to Trillion Parameter Models, Fedus et al., 2021
Sources
- Switch Transformers: Scaling to Trillion Parameter Models, Fedus, Zoph & Shazeer, arXiv:2101.03961, 2021. https://arxiv.org/abs/2101.03961
- DeepSeek-V3 Technical Report, DeepSeek-AI, arXiv:2412.19437, 2024. https://arxiv.org/abs/2412.19437
- DeepSeek-V4-Pro, model card, DeepSeek-AI, Hugging Face (MoE, 1.6 trillion parameters with 49 billion active). https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro
- The Llama 4 herd, Meta AI, April 2025 (first Llama models with an MoE architecture). https://ai.meta.com/blog/llama-4-multimodal-intelligence/
- Qwen3.5-397B-A17B, model card, Alibaba Qwen, Hugging Face (397 billion parameters with 17 billion active). https://huggingface.co/Qwen/Qwen3.5-397B-A17B
- Mistral Large 3, model card, Mistral AI, Hugging Face (granular MoE, 675 billion parameters with 41 billion active). https://huggingface.co/mistralai/Mistral-Large-3-675B-Instruct-2512