Mixture-of-Experts, explained: big-model quality at small-model cost
For most of the history of large language models, choosing a bigger model meant paying for it twice: once in the memory to hold it, and again in the time to run it. Mixture-of-Experts breaks the link between those two costs, and that changes which models are worth serving.
Dense models run everything. Sparse models do not.
In a conventional dense model, generating a token runs the entire network. Every parameter participates in every token, which is why a dense model of twice the size costs roughly twice as much to run as well as twice as much to hold.
A Mixture-of-Experts model replaces some of those layers with a set of parallel expert sub-networks plus a small router that picks a few of them per token. The model still contains all the experts, but any individual token only passes through a handful. The parameters that were not selected sit in memory doing nothing for that token.
That is the mechanism, and its consequence is a model where capacity and compute are decoupled. The Switch Transformer paper, which brought the approach into the mainstream, describes it as a model that "selects different parameters for each incoming example" and summarises the result in a phrase worth keeping:
a sparsely-activated model -- with outrageous numbers of parameters -- but a constant computational cost
Constant computational cost while the parameter count rises is the entire economic argument, and it is why sparsity showed up in open-weight releases the moment serving cost started to matter more than training cost.
The two numbers that matter
Once a model is sparse, its size is no longer one number. You need both, and open-weight MoE releases now name them directly — a model described as 122B-A10B has 122 billion total parameters and about 10 billion active per token.
- Total parameters
- What you must hold in VRAM. Drives your hardware floor: how many cards, how much memory, what quantization you need.
- Active parameters
- What actually runs per token. Drives decode speed, and therefore throughput and cost per million tokens.
A dense model has one number playing both roles. An MoE model lets you buy the first without fully paying for the second — a 122B-A10B model needs memory like a very large model but decodes closer to the speed of a 10B one.
Mixtral was the release that made this concrete for most people, and its own description is the clearest one-line statement of the split:
each token has access to 47B parameters, but only uses 13B active parameters during inference
It reported matching or outperforming Llama 2 70B and GPT-3.5 across the benchmarks it ran. Hold both halves of that in view: the memory bill of a 47B model, the decode speed of a 13B one, and quality claims against models far more expensive to serve than either.
The catch: you still have to hold it
MoE reduces computation, not memory. All those experts have to be resident, which means the hardware floor is set by the total parameter count even though the speed is set by the active count. Sparsity does not let you serve a very large model from a small card.
What it does change is which hardware is the best value. If your constraint is memory capacity rather than compute, then several cheap cards with a lot of aggregate VRAM can beat a single expensive flagship — and cheap high-memory boxes tend to be exactly the ones sitting unused on the spot market.
Quantization stacks with it
Because the constraint MoE leaves you with is memory, quantization is its natural partner: it attacks precisely the axis MoE does not. Serving weights at 4-bit or 8-bit precision instead of 16-bit cuts the memory footprint several-fold, which brings a large sparse model within reach of hardware that could not otherwise hold it.
The combination is what makes big open models practical outside a flagship-GPU budget: sparsity brings the running cost down to something like a small model, quantization brings the memory requirement down to something a modest multi-GPU box can satisfy. Both come with quality risk, which is why neither should be adopted on reputation rather than measurement.
It also matters that the strongest open-weight families now ship first-party quantized builds alongside their full-precision weights. A quantization produced by the people who trained the model is a materially different proposition from one produced by a stranger.
How to evaluate one
- Check the memory arithmetic first. Total parameters × bytes per parameter at your chosen precision, plus headroom for the KV cache. If it does not fit with room to spare, throughput will suffer regardless of how sparse the model is.
- Measure decode throughput on your own hardware, at a concurrency you have actually validated — not the single-stream number in a model card.
- Score it on a gold set built from your real task. Sparsity changes cost; it does not tell you whether the answers are good enough.
- Compare quality per dollar, not quality alone. A slightly weaker model at a third of the cost is often the right answer for batch work.
This is the reasoning behind our own shortlist for the larger product sizes, which leads with sparse candidates: on cheap high-memory hardware, a model with few active parameters is the strongest quality-per-GPU-second lane available. Those remain candidates until they clear an evaluation harness on real workload data — parameter counts do not earn a place in a product tier, measurements do.
Sources
Frequently asked questions
Is an MoE model worse than a dense model of the same total size?
Generally a sparse model performs somewhere between a dense model of its active size and a dense model of its total size, and where it lands varies by model and by task. The useful comparison is not against a dense model of the same total parameters — which would cost far more to run — but against whatever you could otherwise afford to serve.
Do I need multiple GPUs to serve an MoE model?
For the larger ones, usually yes, because the memory requirement follows total parameters. The trade is often favourable: several inexpensive high-memory cards can hold the model and decode quickly because only a fraction of it runs per token.
How do I know how many parameters are active?
Modern open-weight releases put it in the name, in the form total-Aactive — a 35B-A3B model has 35 billion parameters of which roughly 3 billion are active per token. If a model card does not state it, treat the omission as a reason to measure rather than assume.
Related guides
Bigger is not automatically better. A practical method for matching model size and family to a batch task, balancing quality against cost and throughput.
Quantization stores model weights at lower precision, cutting memory and cost. What FP8, INT8 and INT4 mean, what they cost in quality, and which GPUs run them.
The GPU you serve a model on sets both its cost and what it can run. A tour of the T4, L4, L40S, A100 and H100, and how to match a model to the right card.
You cannot choose a model, quantization or prompt without measuring quality. How to build a small gold set and an eval harness that turns choices into data.