Mixture-of-Experts, explained: big-model quality at small-model cost
Can you get a big model’s answers at a small model’s running cost? Partly, with Mixture-of-Experts. A bigger model used to cost you twice: in the memory to hold it, and in the time to run it. A Mixture-of-Experts model breaks the link between those two costs, and that changes which models are worth running.
Dense models run everything. Sparse models do not.
In an ordinary "dense" model, writing each token (a word or part of a word) runs the whole network. Every parameter takes part in every token, so a dense model twice the size costs about twice as much to run, as well as twice as much memory to hold.
A Mixture-of-Experts (MoE) model replaces some of those layers with many smaller "expert" networks side by side, plus a small router that picks a few of them for each token. The model still contains every expert, but each token only passes through a handful. The rest sit in memory, unused for that token.
The result is a model whose size and running cost no longer move together. The Switch Transformer paper, which brought the approach into the mainstream, describes a model that "selects different parameters for each incoming example", and sums up the result in a phrase worth keeping:
a sparsely-activated model -- with outrageous numbers of parameters -- but a constant computational cost
The running cost stays flat while the parameter count rises: that is the whole economic case. It is why these "sparse" models appeared in open releases as soon as the cost of running models began to matter more than the cost of training them.
The two numbers that matter
Once a model is sparse, its size is two numbers, and open MoE releases now put both in the name. A model called 122B-A10B has 122 billion parameters in total, and about 10 billion active for each token.
- Total parameters
- What you must hold in GPU memory. Sets the minimum hardware: how many cards, how much memory, how much compression you need.
- Active parameters
- What actually runs for each token. Sets how fast the model writes, and so the cost per million tokens.
In a dense model one number plays both roles. An MoE model lets you have the first without fully paying for the second: a 122B-A10B model needs the memory of a very large model but writes at close to the speed of a 10-billion-parameter one.
Mixtral was the release that made this concrete for most people, and its own description is the clearest one-line statement of the split:
each token has access to 47B parameters, but only uses 13B active parameters during inference
It reported matching or beating Llama 2 70B and GPT-3.5 on the benchmarks it ran. Keep both halves in view: the memory bill of a 47B model, the writing speed of a 13B one, and quality claims against models far more expensive to run than either.
The catch: you still have to hold it
MoE cuts computation, not memory. Every expert has to be loaded, so the minimum hardware is set by the total parameter count even though the speed is set by the active count. Sparsity does not let you run a very large model on a small card.
What it does change is which hardware is the best value. When memory, not computing power, is what you are short of, several cheap cards with a lot of memory between them can beat one expensive flagship. And cheap high-memory machines are often exactly the ones sitting unused on the spot market, where clouds sell spare capacity at a discount.
Quantization stacks with it
Because MoE leaves memory as the limit, quantization is its natural partner: it cuts exactly the cost MoE does not. Storing the weights in 4 or 8 bits instead of 16 shrinks the memory needed several times over, which brings a large sparse model within reach of hardware that could not otherwise hold it.
Together they make big open models practical without a flagship-GPU budget. Sparsity brings the running cost down to something like a small model’s; quantization brings the memory down to what a modest machine with a few GPUs can hold. Both carry a quality risk, so adopt neither on reputation. Test.
It also helps that the strongest open model families now publish their own compressed versions next to the full-precision weights. A compressed version made by the people who trained the model is a much safer bet than one made by a stranger.
How to evaluate one
- Do the memory sum first: total parameters × bytes per parameter at your chosen precision, plus room for the KV cache (where the model keeps what it has read of each request). If it does not fit with room to spare, it will run slowly however sparse it is.
- Measure how fast it writes on your own hardware, with as many requests at once as you have actually tested, not the single-request number on a model card.
- Score it on a gold set built from your real task. Sparsity changes the cost; it does not tell you whether the answers are good enough.
- Compare quality per dollar, not quality alone. For batch work, a slightly weaker model at a third of the cost is often the right choice.
The same reasoning shapes Otium’s larger sizes, which lean toward sparse models: on cheap high-memory hardware, few active parameters buy the most quality per GPU-second. But a model joins a size only after it passes our own tests on real work. Parameter counts do not earn a place; measurements do.
Sources
Frequently asked questions
Is an MoE model worse than a dense model of the same total size?
A sparse model usually lands somewhere between a dense model of its active size and one of its total size; where exactly varies by model and task. The useful comparison is not with a dense model of the same total size, which would cost far more to run, but with whatever else you could afford to run.
Do I need multiple GPUs to serve an MoE model?
For the larger ones, usually yes, because the memory needed follows the total parameter count. It is often a good trade: several inexpensive high-memory cards can hold the model, and it writes quickly because only a fraction of it runs for each token.
How do I know how many parameters are active?
Recent open releases put it in the name, as total-A-active: a 35B-A3B model has 35 billion parameters, of which about 3 billion are active for each token. If a model card does not say, measure rather than assume.
Related guides
Bigger is not automatically better. How to match model size to a batch task, weigh quality against cost, and send only the hard items to a bigger model.
Quantization stores model weights at lower precision, cutting memory and cost. What FP8, INT8 and INT4 mean, what they cost in quality, and which GPUs run them.
The GPU a model runs on sets both its cost and what it can run. A tour of the T4, L4, L40S, A100 and H100, and how to match a model to the right card.
You cannot choose a model, quantization or prompt without measuring quality. How to build a small gold set and a test loop that turns choices into data.
Several cheap drafts and one model that writes the answer from the best of them. What the research found, when it pays, what it costs, and how to build it.