Learn AI · Level 4 · Frontier view · step 2 of 15 · word 2 of 4
What is a mixture of experts (MoE) model?
A mixture-of-experts model is split into many smaller "expert" parts, and only a few of them are used for each word. That makes a very large model much cheaper to run.
In plain English
Picture a hospital. You don’t see every doctor on every visit. A receptionist sends you to the two specialists you need. In a mixture-of-experts model, a small “router” sends each token to a few experts out of many, and the rest stay idle.
So the model can hold a huge amount of knowledge while each answer uses only a fraction of it.
An example
DeepSeek-V3 has 671 billion parameters in total, but uses only about 37 billion for each token. That’s a big reason it was so cheap to run. Mistral’s Mixtral and many other recent models work the same way.
Watch out
“Experts” is a loose name. They don’t specialise in clear subjects like law or medicine. The whole model still has to fit in memory, so mixture of experts saves computing power, not storage.