A Mixture of Experts (MoE) model contains many specialized sub-networks ("experts"), but only activates a few per token. You get the knowledge of a huge model at the compute cost of a small one.
MoE vs a dense model
This is the key distinction:
| Dense model | MoE model | |
|---|---|---|
| Parameters used per token | All of them | A few experts (sparse) |
| Compute per token | High (scales with size) | Low (scales with active params) |
| Total knowledge capacity | = its size | Much larger than active size |
| Memory (VRAM) | = its size | Still must hold all experts |
A dense 70B model uses all 70B parameters for every token. A 47B-ish MoE like Mixtral 8×7B holds 8 experts but routes each token to just 2 — so it computes like a ~13B model while drawing on far more total knowledge.
How routing works
token → router (gating network) → picks top-2 experts → blend their outputs → next layerA small router learns which experts handle which inputs (code, math, languages…).
The trade-off
MoE buys speed and capacity, but you still need the VRAM to hold every expert even though only a couple run at once. Great throughput, hungry memory.
Related: How Transformers Work · GGUF Quantization Tiers Compared · vLLM vs Ollama vs llama.cpp